The AOS and Donut data flow

Parts

The AOS and Donut processing system is split into four parts.

  • Stamper (donut.Donut) - Takes FITS files as input, and creates “stamps” (cutouts) of donuts from the focus CCDs.
  • Master (donut.Donut) - Takes the stamps written out by the Stamper and assigns them to Workers for processing. It then assembles the results from these workers and passes them along to the AOS
  • Worker/Slave (donut.Donut) - Takes the stamps passed to it by the Master and runs analysis. It then passes the results back to the master and waits for the next stamp. There can be many of these running at a time.
  • AOS (aos.AOS) - Receives the results of Donut analysis, weights the values with bcam data when available/necessary, and sends the results to the hexapod.

Data flow steps

  1. The Stamper scans the image directory looking for files matching the naming convention. This is done by the donut.Donut.file_scan function (running in its own thread). When such a file is found, it is added the list of files to process.
  2. The Stamper’s donut.Donut.file_scan function pops a file off the list, starts a timer that will signal when processing time is up (aossend), and sends the file to the donut.Donut.stamp_single_file function.
  3. The donut.Donut.stamp_single_file function handles control files (files written and deleted to communicate the stage of the process) and passes the file to the donut.Donut.stamp_file function.
  4. The donut.Donut.stamp_file function creates stamps of donuts in the stamp directory from the file given to it. This is done with functions from the donut_stamp.py file.
  5. The Master’s donut.Donut.file_scan function detects stamps being written to the stamp directory. It then takes these files and passes them to the donut.Donut.send_file function to be processed.
  6. The donut.Donut.send_file function updates control files and passes the stamps to the donut.Donut.send_stamp_for_processing function. The donut.Donut.send_stamp_for_processing function prepares the control dictionary for the “feeding” process, and then calls the donut.Donut.feed_donut function.
  7. The donut.Donut.feed_donut function, called by Master, pops a worker off of the queue of available worker roles (self.queue) and makes a PML call to the worker’s donut.Donut.send_donut function, thereby passing the stamp to a worker for processing.
  8. Each Worker has a joblist (self.joblist) that contains the stamps waiting to be processed.* The donut.Donut.send_donut function adds a new job to the list. The donut.Donut.check_jobs function runs repeatedly (triggered by donut.Donut.slave_main), and pulls jobs off of the joblist for processing. It passes these jobs to donut.Donut.process_current_job.
  9. The Worker function donut.Donut.process_current_job sends the job information to the donut_process.py module’s function process_file_full. The results from this analysis are passed back to the donut.Donut.process_current_job function, which in turn passes them back to donut.Donut.check_jobs.
  10. When a stamp has finished processing, the Worker calls donut.Donut.send_master_results, which makes a PML call to donut.Donut.master_results on the Master role.
  11. The master’s donut.Donut.master_results function adds the calling Worker role back to the queue of available Worker roles. It also adds the results dictionary from the analysis to the results list (self.results).
  12. Steps 5-11 are repeated until the Stamper’s timer (aossend) triggers a call to the donut.Donut.send_aos_results function. This function then makes a PML call to the Master’s function donut.Donut.aos_results.
  13. The Master’s donut.Donut.aos_results function calls donut.Donut.build_aos_results, which makes use of the donutana.py and donut_summary.py libraries to construct the results of the analysis. When this is finished, donut.Donut.aos_results makes a PML call to the AOS’s function aos.AOS.donut_results, sending the results to the AOS. It then updates the shared variables and cleans up.
  14. The AOS function aos.AOS.donut_results takes the results from the donut analysis, weights them with values from the bcam if possible, assembles them into a FITS header, and updates shared variables. It also passes data to the function aos.AOS.send_hexapod.
  15. Finally, aos.AOS.send_hexapod runs the values through a PID control, updates the aos_summary SV (indirectly), and passes the new values to the hexapod application via PML.

* Initially the program seemed to support holding more than one job in a worker’s joblist at a time, but this almost never occured due to the worker rejecting new jobs while processing. Although the list structure has been kept, the code has been changed so it signals earlier that it will not accept new jobs, so they aren’t lost in a failed feeding operation by donut.Donut.feed_donut.