Improve License Clarity at Top Package Level

[swastkk]

Fixes #3802

Tasks

• Reviewed contribution guidelines
• PR is descriptively titled 📑 and links the original issue above 🔗
• Commits are in uniquely-named feature branch and has no merge conflicts 📁
• Tests pass -- look for a green checkbox ✔️ a few minutes after opening your PR
• Updated documentation pages (if applicable)
• Updated CHANGELOG.rst (if applicable)
  Run tests locally to check for errors.

Signed-off-by: swastkk swastkk@gmail.com

[AyanSinhaMahapatra]

@swastkk you are regenerating all the tests with useless churn. This adds more things to review and is not ideal. Let's do things differently. You can delete your last commit and push regenerated test fixtures for only the tests which were failing.

[AyanSinhaMahapatra]

@swastkk this is not correct atm:

1. why use license_clarity and not license_clarity_score? This is what we use on the summery option.
2. We want this new attribute license_clarity_score added to top-level packages if and only if the --package-summary option is used, and not in every package like you have here.

This new plugin could be there in packagedcode/plugin_package.py possibly, as we need to check if this option is enabled or not in process_codebase step for the package plugin, and then this should be passed on below to package.to_dict() fucntion for the same.

[AyanSinhaMahapatra]

Does this make sure the license_clarity_score attribute is added only on using the --summary-package command line option? No. Look at your test regenerations, none of these tests have this option enabled, still have this attribute added.

You have to pass the package_summary option like we have the package_only CLI option here: https://github.com/nexB/scancode-toolkit/blob/develop/src/packagedcode/plugin_package.py#L203, then further pass it down to create_package_and_deps at https://github.com/nexB/scancode-toolkit/blob/develop/src/packagedcode/plugin_package.py#L263 and further to package.to_dict() at https://github.com/nexB/scancode-toolkit/blob/develop/src/packagedcode/plugin_package.py#L367 to actually be able to correctly set the attribute package_summary at https://github.com/swastkk/scancode-toolkit/blob/improve-license-clarity/src/packagedcode/models.py#L1544 you added thorugh this PR. Otherwise it's always set to one value.

[AyanSinhaMahapatra]

More changes required.
Please merge latest develop afterwards.

[AyanSinhaMahapatra]

See comments above. And sup on populating license clarity and other attributes?

[AyanSinhaMahapatra]

Couple more nits.

[AyanSinhaMahapatra]

@swastkk See comments and another comment I made above which is still unresolved.

[AyanSinhaMahapatra]

@swastkk you've made changes to the get_field_values_from_resources to send a resources list and not codebase object correctly, but note that we are still not sending anything in score.py: https://github.com/swastkk/scancode-toolkit/blob/improve-license-clarity/src/summarycode/score.py#L78 which would not work effectively if we run only the score option without summary.

You seem to be failing the tests for the same reason, please fix this part. You should be passing all tests to make sure no previous functionality is breaking in this refactor of the compute_license_score to suit the new plugin you're writing.

One minor thing:
The get_codebase_resources seems okay in summarizer.py but we are using dicts: https://github.com/swastkk/scancode-toolkit/blob/improve-license-clarity/src/summarycode/summarizer.py#L323, we should use resource objects here, and also in get_field_values_from_resources we should use resource objects, not list of resource dicts (this means wherever we have resource dicts, we should convert them to Resource objects first and then send in these functions)