Extractor Architecture

The extraction framework is responsible for converting GONet observations into structured records through a collection of independent extractor components.

Each extractor contributes a specific subset of information, such as file metadata, timestamps, weather conditions, astronomical quantities, geometry parameters, or pixel-value measurements.

The final extraction record is assembled by combining the outputs of all participating extractors.

This architecture makes it easy to:

  • Extend the extraction system.

  • Add new output quantities.

  • Reuse extraction logic.

  • Keep extraction code modular and maintainable.

See also:

Key Code Paths

The extraction architecture is implemented across the extractor package and the interactive extraction app.

File or module

Role

GONet_Wizard/GONet_utils/src/extractors/core.py

Base extractor abstractions and dependency declarations.

GONet_Wizard/GONet_utils/src/extractors/runner.py

Orchestrates extractor execution.

GONet_Wizard/GONet_utils/src/extractors/merge.py

Combines extractor result dictionaries into output records.

GONet_Wizard/GONet_utils/src/extractors/file_info.py

Reads file-level metadata and provides shared observation context.

GONet_Wizard/GONet_utils/src/extractors/time_info.py

Adds UTC/local time and MJD fields.

GONet_Wizard/GONet_utils/src/extractors/weather_info.py

Adds weather fields when available.

GONet_Wizard/GONet_utils/src/extractors/astro_info.py

Adds solar and lunar quantities.

GONet_Wizard/GONet_utils/src/extractors/extraction_values.py

Computes channel statistics inside the selected region.

GONet_Wizard/GONet_utils/src/extract_app/shapes/

Defines geometric region classes used by extraction.

Important concepts include USES, PROVIDES, raw, context, extractor result dictionaries, and the final merged JSON/CSV record.

For generated API documentation, see extraction API reference.

Overview

An extraction begins with:

  • One or more GONet observations.

  • Extraction configuration parameters.

  • A collection of extractors.

Depending on the selected extractors, an extraction region may or may not be required.

Each extractor computes a specific subset of the final output.

Conceptually, the extraction pipeline is:

GONetFile
     +
Extraction Region
     |
     v

FileInfo
 │
 └── provides "time"
         │
         ├── TimeInfo
         ├── WeatherInfo
         └── AstroInfo

 ShapeInfo
 ExtractionValues

     |
     v

Merged Output Dictionary

     |
     +--> JSON Output
     |
     +--> CSV Output

The execution order is determined from declared dependencies rather than hard-coded sequencing.

For example, the astronomical, weather, and time extractors all require a valid observation time. That time is provided by the file-information extractor, so the file-information extractor must execute first.

Once the time information is available in the shared context, the dependent extractors may execute.

The final extraction record is produced by merging the output dictionaries returned by each extractor.

Metadata vs Region-Based Extractors

Not every extractor requires an extraction region.

Many extractors operate exclusively on image metadata and can produce useful results without examining any pixels. Examples include the file-information, time, weather, and astronomical extractors.

Only a subset of extractors, such as the extraction-values extractor, require a geometric region in order to compute pixel-based measurements.

The extraction framework therefore supports both metadata-only extraction and region-based measurement workflows.

Why Extractors?

The extractor framework was designed to separate independent concerns that would otherwise accumulate inside a single monolithic extraction routine.

For example:

  • Time conversion logic belongs in the time extractor.

  • Weather retrieval belongs in the weather extractor.

  • Astronomical calculations belong in the astronomical extractor.

  • Pixel statistics belong in the extraction-values extractor.

By isolating these responsibilities, individual components can be developed, tested, and extended independently.

Context-Based Execution

The extraction framework is built around two shared data structures:

raw

Input information supplied to the extraction pipeline.

context

Intermediate information generated by extractors and shared with other extractors.

Every extractor receives both objects:

extract(raw, context)

and returns:

results, context

The results dictionary contributes fields to the final extraction output, while the context dictionary allows extractors to exchange intermediate information.

This design allows extractors to remain largely independent while still supporting complex workflows involving shared computations.

Dependency Example

The standard extraction pipeline contains several extractors that depend on observation timestamps.

Conceptually:

FileInfo
   |
   +--> provides "time"
             |
             +--> TimeInfo
             +--> WeatherInfo
             +--> AstroInfo

The FileInfo extractor reads timestamps from the image files and stores them in the shared context.

The TimeInfo extractor uses those timestamps to generate UTC, local time, and Modified Julian Date fields.

The WeatherInfo extractor uses the same timestamps to retrieve matching weather records.

The AstroInfo extractor uses the same timestamps to compute solar and lunar information.

Without the shared time context value, none of these extractors could operate.

Design Philosophy

The extraction framework follows a composition-based design.

Rather than creating a single monolithic extractor responsible for every output field, the framework divides responsibilities among specialized components.

Each extractor is responsible for one logical category of information.

Benefits of this design include:

  • Simpler code.

  • Easier testing.

  • Better separation of concerns.

  • Straightforward extension of the extraction system.

Extractor Dependencies

Extractors may depend on information generated by other extractors.

Dependencies are declared through two class attributes:

USES

Context keys required by the extractor.

PROVIDES

Context keys generated by the extractor.

For example:

class TimeInfo(Extractor):

    USES = ["time"]
    PROVIDES = []

An extractor may only run once all of its required USES keys are available in the shared context.

Likewise, extractors that generate intermediate context values declare those values through PROVIDES.

These declarations allow the extraction framework to automatically determine a valid execution order.

Note

The shared time value is the most common dependency in the standard pipeline, but the dependency mechanism is not limited to timestamps.

The dependency mechanism is intentionally generic. The framework does not contain any special handling for timestamps or weather information. Any extractor may declare context dependencies, and the dependency resolver will incorporate them into the execution graph automatically.

Any extractor may provide context values that can be reused by other extractors. The framework resolves these relationships automatically using the declared USES and PROVIDES fields.

Automatic Dependency Resolution

The extraction framework does not rely on a manually maintained execution order.

Instead, extractors are topologically sorted according to their declared USES and PROVIDES relationships.

This ensures that:

  • Required context values exist before they are used.

  • New extractors can be added without manually restructuring the pipeline.

  • Dependency errors can be detected automatically.

As a result, the extractor list describes what each extractor needs and provides, while the framework determines when it should execute.

Standard Extractors

The following extractors are included with GONet Wizard.

File Information Extractor

Module:

extractors/file_info.py

Purpose:

  • Filename information.

  • Camera identifier.

  • Unix timestamps.

Typical fields:

  • filename

  • camera

  • unix_time

Shape Information Extractor

Module:

extractors/shape_info.py

Purpose:

Stores the parameters describing the extraction geometry.

Typical fields:

  • shape

  • x0

  • y0

  • radius

  • inner_radius

  • outer_radius

  • start_angle

  • end_angle

Time Information Extractor

Module:

extractors/time_info.py

Purpose:

Converts image timestamps into a variety of useful representations.

Typical fields:

  • night

  • date_utc

  • date_local

  • mjd

  • hours_utc

  • hours_local

  • float_hours_utc

  • float_hours_local

Weather Information Extractor

Module:

extractors/weather_info.py

Purpose:

Retrieves environmental information associated with the observation.

Typical fields:

  • temperature

  • dew_point

  • wind_speed

  • pressure

  • humidity

  • condition_code

Astronomical Information Extractor

Module:

extractors/astro_info.py

Purpose:

Computes astronomical quantities associated with the observation.

Typical fields:

  • sunaltaz

  • moonaltaz

  • moon_illumination

Extraction Values Extractor

Module:

extractors/extraction_values.py

Purpose:

Performs the actual pixel-value measurements within the extraction region.

Typical fields:

  • total_counts

  • mean_counts

  • std

  • npixels

Output Assembly

Each extractor returns a dictionary containing its contribution to the final record.

Conceptually:

{
    "temperature": 27.9,
    "humidity": 54.0,
}

or

{
    "red": {
        "total_counts": ...,
        "mean_counts": ...,
    }
}

The extraction runner combines these dictionaries into a single output record.

The resulting record is then written to either JSON or CSV format.

Output Composition

Each extractor contributes a subset of the final output record.

For example:

FileInfo
   -> filename, camera, unix_time

TimeInfo
   -> date_utc, date_local, mjd

WeatherInfo
   -> temperature, humidity, pressure

ExtractionValues
   -> red, green, blue statistics

The extraction runner merges these contributions into a single dictionary, which is then written to JSON or CSV output.

Extractors generally do not modify fields produced by other extractors. Instead, they contribute additional information to the growing output record.

Results vs Context

A useful way to think about the framework is to distinguish between information intended for the final output and information intended only for intermediate computation.

results

Quantities that will appear in the final extraction record.

context

Intermediate quantities shared between extractors but not necessarily written to the output.

For example, a timestamp may be stored in the context and reused by several extractors, while the final UTC string, local time, and Modified Julian Date are written to the results.

JSON vs CSV

JSON preserves the full hierarchical structure of the extraction results.

Example:

red
  ├── total_counts
  ├── mean_counts
  ├── std
  └── npixels

CSV output flattens nested structures into columns.

Example:

red_total_counts
red_mean_counts
red_std
red_npixels

The two formats contain the same information but are optimized for different use cases.

Minimal Extractor Skeleton

A new extractor should normally look like this:

from GONet_Wizard.GONet_utils.src.extractors.core import Extractor


class MyExtractor(Extractor):
    USES = ["time"]
    PROVIDES = ["my_context_key"]

    def extract(self, raw, context):
        results = {
            "my_output_field": "...",
        }

        context["my_context_key"] = "..."

        return results, context

The runner uses USES and PROVIDES to determine when the extractor can run. The returned results are merged into the final JSON/CSV output, while context carries intermediate values for later extractors.

Adding New Extractors

The framework is intentionally designed to be extensible.

A typical extractor defines:

  • The context keys it requires through USES.

  • The context keys it generates through PROVIDES.

  • An extract() method that computes its results.

When possible, extractors should:

  • Have a single responsibility.

  • Avoid depending on unrelated extractors.

  • Communicate through the shared context rather than direct coupling.

  • Return deterministic outputs for identical inputs.

Following these guidelines helps preserve the modular nature of the framework and keeps dependency graphs easy to understand.

In general, new extractors should focus on a single responsibility and contribute additional fields to the merged output record.

Keeping extractors independent makes the extraction framework easier to test, maintain, and extend.

Relationship with the API Reference

This page describes the architecture of the extraction framework.

For implementation details and class documentation, see:

For user-facing extraction workflows, see: