(redirected from DNRNotes.MondayMeeting)

Doug's December 2^nd-7^th 2007 Visit to DNR

Topics for discussion
Monday Meeting in CBI Conference Room
Tuesday Meeting Agenda
Results from the Tuesday and Thursday Meetings

Results from the Tuesday and Thursday Meetings

December 4 and 6, 2007 Attendees: James D., James R., Doug, Niall

Prioritized Tasks

1. Gap Filling Tools - Ability to insert data and draw curves and such.
2. Automated Data Recovery - Restrict to Sutron Xpert and IP modems only. James R. will give us a list of stations we can test this on.

JR and his Team will implement manual (PC, telephone or station visit) downloads monthly until automated recovery is functional.

1. Benchmark Analysis Tool

Ongoing Task

• User Guide will require everyone's assistance and update-as-you-go to make sure it stays up to date w/ the Pharos web interface.

Additional Items

• Can we push Doug's processing date back from the 1^st of each month? This would give us additional time to ensure that all collectible data is available in the system before processing. Otherwise, the processing will need to be redone when all collected data is available.

Is five working days enough time before starting processing?

• Implement data verification protocol where field crew personnel (Drew, Richard) enter benchmark information into the system themselves (instead of having Sara do it) and Sara looks over the data to double-check the entries and look for inconsistencies.
• Auto-importing of .abs and .in files? What can be done to further this task? Is manual modification/human intervention necessary because of re-use of SPSNs?
• Review/update comparison stations as the list is out of date.
• We need to show there is benefit to the State of Texas; private, local, state and federal organizations are using our data.

Tuesday Meeting Agenda

With Doug

• Doug's Processing of Monthly Means
  • Watch Doug doing this
  • What should be automated?
  • Formatting of output?

• Filling Data Gaps
  • Clearly delineate (generated vs. measured)
  • How/where to store gap fills
  • Store how the gap was filled (method, input, when, who), not the fill. Generate the fill data on the fly?

• Automated Data Recovery
  • GOES/NESDIS - lrgs disconnect? Telemetry issues we can't control.
  • More robust/automated data recovery
  • IP modem data recovery
  • How to handle loss of Internet connection

• Benchmark Analysis Tool
  • Do it!

• Do We Enlist Pm To Help Out?
  • User guide? Use Pm or have James/I do it? When does it need to be completed?
  • Time/Manpower tradeoff?
  • Look at it
  • How much more does Doug think needs to be done?

James and Niall

• Breakdown tasks
• Come up with time estimates
• Submit to Gary

Monday Meeting

December 3rd, 2007 CBI Conference Room 10am-12pm Attendees: Doug, James R., James D., John, Gary, and Niall

Niall's Random Notes Throughout The Meeting

We need more documentation for tracking of data and training purposes. Think of the chain of evidence in a courtroom case.

CO-OPS is fully automated w/ batch processing of data, we are not.

Monthly Product Generator - automated production of monthly means summary, submitted to human (analysis) for approval.

We need the batch processing, right now its initiated by a human. You have to process on a calendar basis, by the first of the month.

Doug processes everything in the first week of the month.

Do we need Pat to come in and work on the batch processing task and/or user guide?

We need editing and break-filling tools similar to CO-OPS. We could improve the grades on months from C to B (then we can use those months in the datum calculation).

Only Grade A's and B's are used in datum computation, so if we improve the lower grades to a B or better, we have more data for the datum computation, and of course, the database is improved overall.

CO-OPS has a hierarchy of how to fill a break:

1. If they have backup data, they use that to fill
2. Fill w/ a nearby station. (when filling high and low water break)
3. If no high/low water break, use a linear, least squares or spline to fill gap

Questions about data gaps. How much do the gaps influence the monthly mean? How much data can we miss w/o it affecting the monthly mean?

Improving our data product, improving the website (James' contact).

Dr. Jeffreys wants up-to-date sea level for Padre Island (flood insurance and compliance). Something that calculates up-to-date sea levels?

Flood elevations? Flood maps? Boundary purposes.

Doug wants 5 years of monthly mean data in a single shot. Present output is not useful. Spreadsheet output.

Sounds like Doug wants specialized output (calendar monthly interval instead of daily) with a few extra series (annual means, 5 year mean, 19 year mean) for the stations he is calculating datums.

Benchmark analysis Excel program? Implement that on our system.

Conclusions

Tasks and/or things to investigate further:

• Modify data query to support calendar month intervals, yearly averages (water level), 5 year averages, etc. Automate the queries for Doug so he can use them when performing water level datum tabulations.
• We discussed gap filling, automated data recovery, using predictions (Philippe's ANNs), etc. I don't think this is a reasonable short-term task.
• Data verification protocol? Documentation? What does this mean?

Verification of processed data means to have another person look over the data to ensure accuracy.

• Implement Bench Mark Analysis tool on our system (using Excel spreadsheet from Doug as a guide).

This analysis tool was provided to me by CO-OPS with their permission to use as a development guide for CBI.

• Look at Doug's user guide. Call in Pm to work on guide or do it ourselves?
• Evaluate these tasks, document them, determine time requirements and consult w/ Gary on them.
• Submit to Gary these tasks to help justify hiring a new IT person, come up w/ 1 year-long list of tasks for this purpose.

Topics for discussion:

• Current processing and automated batch processing of water level (tide) data for TCOON.
• Implementing some editing and break filling tools similar to CO-OPS.
• Establishing a data verification protocol.
• Implementing the Bench Mark Analysis Excel program provided by CO-OPS.
• Look at present procedures for entering bench mark elevations compared to CO-OPS.
• Discuss the disposition of the draft User's Guide
• Special procedures for 5 year datum updates and what we should do with TCOON datums.
• Federal Information Security Management Act (FISMA).

Gary - available up until Thursday
James R - available a few days
James D - available all week
Niall -