Merge Develop into Release (#3179)

* add Infacost rule (#3164)

* New Published Rules - returntocorp.reserved-aws-lambda-environment-variable (#3159)

* add returntocorp/reserved-aws-lambda-environment-variable.yaml

* add returntocorp/reserved-aws-lambda-environment-variable.tf

* move files

---------

Co-authored-by: Grayson H <grayson@semgrep.com>
Co-authored-by: Vasilii <inkz@xakep.ru>

* Update `server-dangerous-object-deserialization` rule

I'm making a change to the Semgrep Pro Engine which will correctly model
the fact that `java.lang.Object` is at the top of any inheritance
hierarchy in Java. As such, the pattern `Object $X` will match any
object type, including the `String` types used in the tests here.

In general, we do want the Pro Engine, when presented with a pattern
`(Foo $X)`, to also match any subtypes of `Foo`.

Based on a discussion with Pieter, this should actually match any object
type except for `String`s and boxed types. As such, I have updated this
rule and the test cases accordingly. Now, it functions the same both on
the Pro Engine and OSS, and is more accurate than before on both.

* Add rule for missing depends_on in subscription filters shipping to lambdas (#3168)

* Add rule flagging redundant fields on AWS Lambda resource when using Image package_type (#3167)

* Add rule flagging redundant fields on AWS Lambda resource when using Image package_type

* Shorten rule ID a tad

---------

Co-authored-by: Pieter De Cremer (Semgrep) <pieter@r2c.dev>

* Updated unverified-jwt-token according to new APIs. Added fixtest (#3170)

* Updated unverified-jwt-token according to new APIs. Added fixtest

* move test syntax to correct line

* Update test targets for tests relying on include:

Thanks to semgrep/semgrep#8993
the include: directive in the rule is now ignored in a test context,
so you can use back the same name than the rule for the test target
file

test plan:
see related PR in semgrep

* Add rule for missing asterisk at end of aws_lambda_permission cloudwatch permissions (#3163)

* Add rule for missing asterisk at end of aws_lambda_permission cloudwatch permissions

* Remove swap file, add tech metadata

* Shorten rule ID a tad

* Skip Apex rules when running the OSS testsuite (#3177)

* Skip Apex rules in testsuite - part 2 (#3178)

* Skip Apex rules when running the OSS testsuite

* Skip Apex rules in testsuite - part 2

---------

Co-authored-by: Lewis <LewisArdern@live.co.uk>
Co-authored-by: semgrep-dev-pr-bot[bot] <63393893+semgrep-dev-pr-bot[bot]@users.noreply.github.com>
Co-authored-by: Grayson H <grayson@semgrep.com>
Co-authored-by: Vasilii <inkz@xakep.ru>
Co-authored-by: Nat Mote <nat@natmote.net>
Co-authored-by: Nat Mote <nat@semgrep.com>
Co-authored-by: Pieter De Cremer (Semgrep) <pieter@r2c.dev>
Co-authored-by: pad <pad@r2c.dev>
Co-authored-by: Claudio <claudio@r2c.dev>

.github/workflows/semgrep-rules-test-develop.yml

@@ -22,6 +22,8 @@ jobs:
       run: rm -rf semgrep-rules/stats
     - name: delete fingerprints directory
       run: rm -rf semgrep-rules/fingerprints
+    - name: delete rules requiring Semgrep Pro
+      run: rm -rf semgrep-rules/apex
     - name: validate rules
       run: |
         export SEMGREP="docker run --rm -w /src -v ${GITHUB_WORKSPACE}/semgrep-rules:/src returntocorp/semgrep:develop semgrep"

.github/workflows/semgrep-rules-test-historical.yml

@@ -34,6 +34,8 @@ jobs:
       run: rm -rf semgrep-rules/stats
     - name: delete fingerprints directory
       run: rm -rf semgrep-rules/fingerprints
+    - name: delete rules requiring Semgrep Pro
+      run: rm -rf semgrep-rules/apex
     - name: grab historical semgrep version
       env:
         GH_TOKEN: ${{ github.token }}

.github/workflows/semgrep-rules-test.yml

@@ -23,6 +23,8 @@ jobs:
       run: rm -rf stats
     - name: remove fingerprints from testing
       run: rm -rf fingerprints
+    - name: remove rules requiring Semgrep Pro
+      run: rm -rf apex
     - name: validate rules
       run: semgrep --validate --config .
     - name: run semgrep

python/jwt/security/unverified-jwt-decode.fixed.py

@@ -0,0 +1,33 @@
+# cf. https://github.com/we45/Vulnerable-Flask-App/blob/752ee16087c0bfb79073f68802d907569a1f0df7/app/app.py#L96
+
+import jwt
+from jwt.exceptions import DecodeError, MissingRequiredClaimError, InvalidKeyError
+
+def tests(token):
+    # ruleid:unverified-jwt-decode
+    jwt.decode(encoded, key, options={"verify_signature": True})
+
+    # ruleid:unverified-jwt-decode
+    opts = {"verify_signature": True}
+    jwt.decode(encoded, key, options=opts)
+
+    a_false_boolean = False
+    # ruleid:unverified-jwt-decode
+    opts2 = {"verify_signature": True}
+    jwt.decode(encoded, key, options=opts2)
+
+    # ok:unverified-jwt-decode
+    jwt.decode(encoded, key, options={"verify_signature": True})
+
+    opts = {"verify_signature": True}
+    # ok:unverified-jwt-decode
+    jwt.decode(encoded, key, options=opts)
+
+    a_false_boolean = True
+    opts2 = {"verify_signature": a_false_boolean}
+    # ok:unverified-jwt-decode
+    jwt.decode(encoded, key, options=opts2)
+
+    # ok:unverified-jwt-decode
+    jwt.decode(encoded, key)
+

python/jwt/security/unverified-jwt-decode.py

@@ -3,21 +3,31 @@
 import jwt
 from jwt.exceptions import DecodeError, MissingRequiredClaimError, InvalidKeyError
 
-def verify_jwt(token):
-    try:
-        # ok:unverified-jwt-decode
-        decoded = jwt.decode(token, app.config['SECRET_KEY_HMAC'], verify=True, issuer = 'we45', leeway=10, algorithms=['HS256'])
-        print("JWT Token from API: {0}".format(decoded))
-        return True
-    except DecodeError:
-        print("Error in decoding token")
-        return False
-    except MissingRequiredClaimError as e:
-        print('Claim required is missing: {0}'.format(e))
-        return False
-
-def insecure_verify(token):
+def tests(token):
     # ruleid:unverified-jwt-decode
-    decoded = jwt.decode(token, verify = False)
-    print(decoded)
-    return True
+    jwt.decode(encoded, key, options={"verify_signature": False})
+
+    # ruleid:unverified-jwt-decode
+    opts = {"verify_signature": False}
+    jwt.decode(encoded, key, options=opts)
+
+    a_false_boolean = False
+    # ruleid:unverified-jwt-decode
+    opts2 = {"verify_signature": a_false_boolean}
+    jwt.decode(encoded, key, options=opts2)
+
+    # ok:unverified-jwt-decode
+    jwt.decode(encoded, key, options={"verify_signature": True})
+
+    opts = {"verify_signature": True}
+    # ok:unverified-jwt-decode
+    jwt.decode(encoded, key, options=opts)
+
+    a_false_boolean = True
+    opts2 = {"verify_signature": a_false_boolean}
+    # ok:unverified-jwt-decode
+    jwt.decode(encoded, key, options=opts2)
+
+    # ok:unverified-jwt-decode
+    jwt.decode(encoded, key)
+

python/jwt/security/unverified-jwt-decode.yaml

@@ -1,7 +1,25 @@
 rules:
 - id: unverified-jwt-decode
-  pattern: |
-    jwt.decode(..., verify=False, ...)
+  patterns:
+    - pattern-either:
+        - patterns:
+          - pattern: |
+              jwt.decode(..., options={..., "verify_signature": $BOOL, ...}, ...)
+          - metavariable-pattern:
+              metavariable: $BOOL
+              pattern: |
+                False
+          - focus-metavariable: $BOOL
+        - patterns:
+            - pattern: |
+                $OPTS = {..., "verify_signature": $BOOL, ...}
+                ...
+                jwt.decode(..., options=$OPTS, ...)
+            - metavariable-pattern:
+                metavariable: $BOOL
+                pattern: |
+                  False
+            - focus-metavariable: $BOOL
   message: >-
     Detected JWT token decoded with 'verify=False'. This bypasses any integrity
     checks for the token which means the token could be tampered with by
@@ -24,9 +42,8 @@ rules:
     likelihood: MEDIUM
     impact: MEDIUM
     confidence: MEDIUM
-  fix-regex:
-    regex: (verify\s*=\s*)False
-    replacement: \1True
+  fix: |
+    True
   severity: ERROR
   languages:
   - python

scripts/run-tests

@@ -40,9 +40,12 @@ fi
 # may contain .yml files that are not Semgrep rules and would result
 # in errors.
 #
+# Skipping the "Apex" folder because it will require splitting test logic
+# to run Semgrep OSS and Semgrep Pro with different expected results.
+#
 set_rule_folders() {
   rule_folders=$(find . -mindepth 1 -maxdepth 1 -type d \
-    | grep -v '^./\(\..*\|stats\|trusted_python\|fingerprints\|scripts\|libsonnet\)/\?$' \
+    | grep -v '^./\(\..*\|stats\|trusted_python\|fingerprints\|scripts\|libsonnet\|apex\)/\?$' \
     | sort)
   if [[ -z "$rule_folders" ]]; then
     error "Cannot find any rule folders to scan in $(pwd)"

terraform/aws/correctness/lambda-permission-logs-missing-arn-asterisk.tf

@@ -0,0 +1,31 @@
+resource "aws_lambda_permission" "allow_cloudwatch" {
+  statement_id  = "${var.name}-allow-execution-from-cloudwatch"
+  action        = "lambda:InvokeFunction"
+  function_name = aws_lambda_function.lambda_function.function_name
+  principal     = "logs.amazonaws.com"
+  # ruleid: lambda-permission-logs-missing-arn-asterisk
+  source_arn    = "arn:aws:logs:us-west-2:${data.aws_caller_identity.current.account_id}:log-group:${var.log_group_name}"
+
+  depends_on = [aws_lambda_function.lambda_function]
+}
+
+resource "aws_lambda_permission" "allow_cloudwatch" {
+  statement_id  = "${var.name}-allow-execution-from-cloudwatch"
+  action        = "lambda:InvokeFunction"
+  function_name = aws_lambda_function.lambda_function.function_name
+  principal     = "logs.amazonaws.com"
+  # ok: lambda-permission-logs-missing-arn-asterisk
+  source_arn    = "arn:aws:logs:us-west-2:${data.aws_caller_identity.current.account_id}:log-group:${var.log_group_name}:*"
+
+  depends_on = [aws_lambda_function.lambda_function]
+}
+
+resource "aws_lambda_permission" "allow_cloudwatch" {
+  statement_id  = "AllowExecutionFromCloudWatch"
+  action        = "lambda:InvokeFunction"
+  function_name = aws_lambda_function.test_lambda.function_name
+  principal     = "events.amazonaws.com"
+  # ok: lambda-permission-logs-missing-arn-asterisk
+  source_arn    = "arn:aws:events:eu-west-1:111122223333:rule/RunDaily"
+  qualifier     = aws_lambda_alias.test_alias.name
+}

terraform/aws/correctness/lambda-permission-logs-missing-arn-asterisk.yaml

@@ -0,0 +1,25 @@
+rules:
+- id: lambda-permission-logs-missing-arn-asterisk
+  severity: WARNING
+  languages: [hcl]
+  message: "The `source_arn` field needs to end with an asterisk, like this: `<log-group-arn>:*` Without this, the `aws_lambda_permission` resource '$NAME' will not be created. Add the asterisk to the end of the arn. x $ARN"
+  metadata:
+    category: correctness
+    references:
+    - https://github.com/hashicorp/terraform-provider-aws/issues/14630
+    technology:
+    - aws
+    - terraform
+    - aws-lambda
+  patterns:
+    - pattern-inside: |
+        resource "aws_lambda_permission" "$NAME" { ...  }
+    - pattern: |
+        source_arn = $ARN
+    - metavariable-pattern:
+        metavariable: $ARN
+        patterns:
+        - pattern-regex:
+            arn:aws:logs.*
+        - pattern-not-regex: >-
+            arn:aws:logs:.*:\*

terraform/aws/correctness/lambda-redundant-field-with-image.tf

@@ -0,0 +1,112 @@
+resource "aws_lambda_function" "forward_to_sns" {
+  function_name = "${var.name}-cloudwatch-forward-to-sns"
+  role          = aws_iam_role.lambda_to_sns.arn
+  timeout       = 120
+  package_type  = "Image"
+  image_uri     = "image/goes/here"
+  # ruleid: lambda-redundant-field-with-image
+  handler       = "main.lambda_handler"
+  # ruleid: lambda-redundant-field-with-image
+  runtime       = "python3.9"
+
+  environment {
+    variables = {
+      SNS_TOPIC_ARN           = aws_sns_topic.sns_topic.arn
+      AWS_REGION_OF_SNS_TOPIC = var.region
+    }
+  }
+  depends_on = [aws_iam_role.lambda_to_sns]
+}
+
+resource "aws_lambda_function" "forward_to_sns" {
+  function_name = "${var.name}-cloudwatch-forward-to-sns"
+  role          = aws_iam_role.lambda_to_sns.arn
+  timeout       = 120
+  package_type  = "Image"
+  image_uri     = "image/goes/here"
+  # ruleid: lambda-redundant-field-with-image
+  handler       = "main.lambda_handler"
+
+  environment {
+    variables = {
+      SNS_TOPIC_ARN           = aws_sns_topic.sns_topic.arn
+      AWS_REGION_OF_SNS_TOPIC = var.region
+    }
+  }
+  depends_on = [aws_iam_role.lambda_to_sns]
+}
+
+resource "aws_lambda_function" "forward_to_sns" {
+  function_name = "${var.name}-cloudwatch-forward-to-sns"
+  role          = aws_iam_role.lambda_to_sns.arn
+  timeout       = 120
+  package_type  = "Image"
+  image_uri     = "image/goes/here"
+  # ruleid: lambda-redundant-field-with-image
+  runtime       = "python3.9"
+
+  environment {
+    variables = {
+      SNS_TOPIC_ARN           = aws_sns_topic.sns_topic.arn
+      AWS_REGION_OF_SNS_TOPIC = var.region
+    }
+  }
+  depends_on = [aws_iam_role.lambda_to_sns]
+}
+
+resource "aws_lambda_function" "forward_to_sns" {
+  function_name = "${var.name}-cloudwatch-forward-to-sns"
+  role          = aws_iam_role.lambda_to_sns.arn
+  timeout       = 120
+  package_type  = "Image"
+  # ok: lambda-redundant-field-with-image
+  image_uri     = "image/goes/here"
+
+  environment {
+    variables = {
+      SNS_TOPIC_ARN           = aws_sns_topic.sns_topic.arn
+      AWS_REGION_OF_SNS_TOPIC = var.region
+    }
+  }
+  depends_on = [aws_iam_role.lambda_to_sns]
+}
+
+resource "aws_lambda_function" "forward_to_sns" {
+  function_name = "${var.name}-cloudwatch-forward-to-sns"
+  role          = aws_iam_role.lambda_to_sns.arn
+  timeout       = 120
+  package_type  = "Zip"
+  image_uri     = "image/goes/here"
+  # ok: lambda-redundant-field-with-image
+  handler       = "main.lambda_handler"
+  # ok: lambda-redundant-field-with-image
+  runtime       = "python3.9"
+
+  environment {
+    variables = {
+      SNS_TOPIC_ARN           = aws_sns_topic.sns_topic.arn
+      AWS_REGION_OF_SNS_TOPIC = var.region
+    }
+  }
+  depends_on = [aws_iam_role.lambda_to_sns]
+}
+
+
+resource "aws_lambda_function" "forward_to_sns" {
+  function_name = "${var.name}-cloudwatch-forward-to-sns"
+  role          = aws_iam_role.lambda_to_sns.arn
+  timeout       = 120
+  image_uri     = "image/goes/here"
+  # ok: lambda-redundant-field-with-image
+  handler       = "main.lambda_handler"
+  # ok: lambda-redundant-field-with-image
+  runtime       = "python3.9"
+
+  environment {
+    variables = {
+      SNS_TOPIC_ARN           = aws_sns_topic.sns_topic.arn
+      AWS_REGION_OF_SNS_TOPIC = var.region
+    }
+  }
+  depends_on = [aws_iam_role.lambda_to_sns]
+}

terraform/aws/correctness/lambda-redundant-field-with-image.yaml

@@ -0,0 +1,23 @@
+rules:
+- id: lambda-redundant-field-with-image
+  severity: WARNING
+  languages: [hcl]
+  message: 'When using the AWS Lambda "Image" package_type, `runtime` and `handler` are not necessary for Lambda to understand how to run the code. These are built into the container image. Including `runtime` or `handler` with an "Image" `package_type` will result in an error on `terraform apply`. Remove these redundant fields.'
+  metadata:
+    category: correctness
+    references:
+    - https://stackoverflow.com/questions/72771366/why-do-i-get-error-handler-and-runtime-must-be-set-when-packagetype-is-zip-whe
+    technology:
+    - aws
+    - terraform
+    - aws-lambda
+  patterns:
+  - pattern-inside: |
+      resource "aws_lambda_function" $NAME { 
+        ...
+        package_type = "Image"
+      }
+  - pattern-either:
+      - pattern: handler = ...
+      - pattern: runtime = ...
+