Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug: GET /v1/pipelines should include associated processors #2067

Open
raulb opened this issue Jan 10, 2025 · 0 comments
Open

Bug: GET /v1/pipelines should include associated processors #2067

raulb opened this issue Jan 10, 2025 · 0 comments
Labels
bug Something isn't working triage Needs to be triaged

Comments

@raulb
Copy link
Member

raulb commented Jan 10, 2025

Bug description

Important

Fixing this issue would be nice since we could include the processors as part of conduit pipeline ls before we release 0.13.0

Working on #2019, I noticed that even though a pipeline contains a processor, the PipelineService doesn't include this information, appearing to be empty:

curl -X 'GET' \
  'http://localhost:8080/v1/pipelines' \
  -H 'accept: application/json'  
[
  {
    "id": "add-department",
    "state": {
      "status": "STATUS_RUNNING",
      "error": ""
    },
    "config": {
      "name": "add-department",
      "description": "An example pipeline which reads data (imaginary employees) from two generator sources, processes it and writes it to a file.\nIt attaches the built-in `field.set` processor to one of the sources to add a `department` field to its records. The records from the other source are not processed.\n"
    },
    "connectorIds": [
      "add-department:employees-1",
      "add-department:employees-2",
      "add-department:file-destination"
    ],
    "processorIds": [],
    "createdAt": "2025-01-09T12:23:13.737117Z",
    "updatedAt": "2025-01-09T12:23:13.737693Z"
  }
]

Here's my yaml

version: 2.2
pipelines:
  - id: add-department
    status: running
    description: >
      An example pipeline which reads data (imaginary employees) from two generator
      sources, processes it and writes it to a file.
      
      It attaches the built-in `field.set` processor to one of the sources
      to add a `department` field to its records. The records from the other source
      are not processed.
    connectors:
      - id: employees-1
        type: source
        plugin: builtin:generator
        settings:
          format.type: "structured"
          format.options.id: int
          format.options.name: string
          format.options.company: string
          format.options.trial: bool
          recordCount: "1"
        processors:
          - id: extract-name
            plugin: field.set
            settings:
              field: '.Payload.After.department'
              value: 'finance'
      - id: employees-2
        type: source
        plugin: builtin:generator
        settings:
          # department collection
          format.type: "structured"
          format.options.id: int
          format.options.name: string
          format.options.company: string
          format.options.trial: bool
          recordCount: "2"
      - id: file-destination
        type: destination
        plugin: builtin:file
        settings:
          path: ./example.out

Steps to reproduce

  1. Make sure you have a pipeline with a processor.
  2. Run conduit
  3. Visit http://localhost:8080/openapi/#/PipelineService/PipelineService_ListPipelines and inspect the response on this request.
  4. Notice processors are not included on the pipeline that has it.

Version

v0.13.0-nightly.20250110

@raulb raulb added bug Something isn't working triage Needs to be triaged labels Jan 10, 2025
@raulb raulb mentioned this issue Jan 10, 2025
36 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working triage Needs to be triaged
Projects
Status: Triage
Development

No branches or pull requests

1 participant