Request for Comments

Items mentioned here will be coalesced into tasks to be done in Nimble

What's wrong with our software or hardware? What would you like to see changed? Add to the list below:

• Replace sendmail with postfix or qmail
• Old code needs to be modernized.
• We should use PDL for calculating high/low tides
• "modularize" things that should be modules from pharosdb
• Website organization and navigation
• How are people using our website at the moment? User testing/monitoring? Google Analytics
• Develop website "customer" profiles
• Measuring speed, finding bottlenecks in pharosdb data retrieval/presentation
• Mutual Interruption Shields
• Accurately track data requests and break down based on station, series, time period, determine which request(ers) are robots
• Network security analysis - are we encrypting all sensitive traffic (LDAP, IMAP, POP authentications, LDAP DB sync, etc.)? Are we doing anything extremely stupid/easy to exploit?
• Remove sensitive information (passwords) from pharosdb repository
• Add Geo IP lookup to initially center station map on web visitor's location
• Make programmatic portions of the website more MVCish

• where to incorporate google analytics?

Enhanced data display

• tinyurl functionality added to pq,puqi, or where ever else it is appropriate
• rethink the tabs across the top of the station pages in light of many non-TCOON stations now and in the future
• where to put "temporary" stations? (There's no such thing. Stations are immortal once created. Rather than segregating based on ID, they should just use the next available ID and use the class mechanism to group stations)
• mechanism to modify URL parameters (when a user only has a URL, make it such that they can change the date range, station, etc.)
• Use a nautical chart ala google maps
• put wind, water current, temperature, barometric pressure, and station location on map
• get pictures of common landmarks in the area and show them on the map.
• relate water level to common landmarks
• Implement timeline

Ideas from Pm

• get a small, cheap Dell notebook and measure the real power draw. Might be useful for a data-collector in an emergency
• Embed parrot on data collector
• put webserver on data collector
• web-based configuration for data collector

System/Information Security

• Implement user ACLs (using ldap information) so that sponsor X cannot see sponsor Y's :private: stations/download data

John Adams' 2009 Wish List

From John's Friday, 9 January 2009 e-mail:

• http://lighthouse.tamucc.edu/Salinity/HomePage - i would like to see stations 069, 011, and 153 added to this interactive page. different symbols for 069 (weather stations), 135 & 153 are tide stations. the legend would tell the user what type of station that are looking at.
• http://lighthouse.tamucc.edu/GBRA/HomePage - this should be an interactive map just like all the others. stations 127 and 031.
• http://lighthouse.tamucc.edu/RTNS/HomePage - get rid of the reference to freeport flow info. have the homepage go directly to http://lighthouse.tamucc.edu/RTNS/CorpusChristiRTNS add 006, 009, 008, 135. and 108 to the map. different symbols for 006, 008, & 009 (tide gauges) different symbols for 135 & 108 (weather stations)
• http://lighthouse.tamucc.edu/MissionAransas/HomePage - add the two tide stations 036 & 015 (different symbols) add the GBRA (127) if it fits (different symbol as explained in the legend)
• http://lighthouse.tamucc.edu/BahiaGrande/HomePage - this map needs to be larger so the user can tell which part of the coast they are looking at. if the map is expanded enough, add the port isabel (018) and south padre tide station(051)
• a new interactive map showing 170 & 171 in the laguna madre. baffin (0680 and bid (013) should be shown also.
• updates and progress on the data logger.

[sic]

Doug's March 2009 Visit

From Doug's Friday, 6 March 2009 e-mail:

Per our conversations and meetings this week, we have identified three tasks that will improve TCOON data processing and also improve overall data quality. Those tasks are as follows:

1. Automated data recovery.
2. Data editing tools i.e. spike identification and gap filling.
3. Bench mark stability analysis tools.

Please contact me if you have any questions during the development and implementation of these tasks.

This task list is out of date and needs to be edited.

Here are some items that DNR would like to see done. These items are designed such that they can be used as student projects.

Easy projects that can be done relatively quickly.

• Fix illus.cgi to do something useful when given multiple stations.
• Fix illus.cgi to show the well even when datums aren't available.
• A program that will be run nightly to "prune" the list of IP addresses we've seen in ~pharosdb/var/survey-ips. See ~pharosdb/lib/Survey.pm for how to manipulate this file as a hash in perl. The values stored are the time at which we saw the particular IP. IPs^? that are > 30 * 86400 seconds old should be pruned.
• Write a program to collect the information given by the user surveys and generate a report.
• A CGI program that will read the survey data collected in ~pharosdb/var/survey/* and generate a report of the results. This program should accept a "when" parameter to limit the range of dates over which the report will be generated.
• automatically send email when certain station parameters are updated. The default email address should be dnr-field@lighthouse.tamucc.edu. The default list of attributes to "watch" are c1c2,plid,flid.
• Write a decoder for Philippe's data (stage height from St. John's River in Florida and water level data from somewhere in Port Aransas) (2)
• Change cachelatest such that it can be used for "Recent Observations" in overview.cgi
• Modify stnmap to format latest observation popup as a list (and change CSS to make it look nice)
• write a program to see which stations have wsd/wgt but do not have a correction record to limit the range to 0-50 m/s
• Write a program to automatically create appropriate correction records for wsd/wgt

Harder projects

These need further thought (If you want to work on one of these call a meeting of dnr-is@lighthouse.tamucc.edu) Also, these and the hardest projects may require pmwiki2.

• create an area for field crew to upload/download leveling files (2)
• create an interface for field crew to upload pictures of stations and other associated images and automatically organize these files as much as possible.(2)

Hardest projects

• Organize the DNR website (1)
• Wiki TCOON Quarterly reports (1)
  • use pmwiki2
  • use recipe for publishing wikitrails to pdf
• Incorporate NWS wind models into DNR database (3)
• replace VERTPGM with a perl script (5)
• Improved visualization of wind data (as time-series in 2 dimensions, primarily wind direction)
• Better visualization of DNR network status (tide gauge network, not ethernet)
  • Where are the problems?
  • How severe are the problems?
  • tied to metadata about who to contact or what to do? help system?
• Find a way to automatically make sure the voice data systems are working and giving good data
• Update PHP programs for system administration tasks to work with pmwiki

Other cool things

• Use the Beowulf cluster to build an evolutionary framework for creating neural networks that predict water level using perl + PDL (Scott's pet project)
• Build a generalized neural network framework in Perl+PDL (Scott's other pet project)
• Parallelize neural network training using the Beowulf cluster (James' pet project)
• finish pd.cgi
  • incorporate lrwl.pm
  • create nnwl.pm
  • documentation
  • convert all of ~pharosdb/cgi-bin/* to use pd.cgi or PSeries
• split PSeries stuff out of pd.cgi

These next two items are different aspects of the same thing

• design and implement a better "messaging system" for QCs^?
• design and implement a better priority todo list

• create smarter replacement for pquery (1 duff, see PQueryNotes)
• automated emails to NOAA when plid,flid,c1, or c2 changes on selected stations (cron job ?)
• Alert to flag when the last time O&M was done at a station.
• user tracking of DNR data requests
• personalized DNR experience (using LDAP?)
• setup accounts for people who use the DNR site.
• "Inventory tracking system" Something to allow personnel to query sitevisits for serial numbers (what instrument is deployed where)
• modify correction records so that they can persist indefinitely into the future.
• Create mechanism for field crew to selectively suppress data series on a per-station, per-series, per-time-range basis.
• rewrite vertpgm (needs to be done before Sept 1)

• Make vlgscan more robust to poorly formed input
  • currently dies if the pwl value looks like it should be english rather than metric for even one line only. It should sanity check the line before getting to that point.
• Wiki quarterly reports
• form for putting leveling files in dnrpub
• picture uploads for field crew
• leveling files available from bmarks.cgi (3, mostella)
• Joint TABS/TCOON web site for background/policy/political information (2, duff/jeffress)
  • Need page design
  • Need person to do the page design (1, pmichaud)
  • Need content for new site

• make gumstix computer do data collection (3)
• Explain the monthly grading system on website somewhere
• Create a page comparing/displaying varying water level prediction models (4?, mostella)
• Software Documentation (2, mostella)

• migrate to pmwiki2
• Explore using Festival to generate voice output for voiceinfo
• tcoon.org?
• Change default parameters for /stnlist
• spike removal filter for pwl
• Add NDBC stations to DNR database
• Fill out WLModelComparisons
• Begin documenting different prediction and evaluation methods for NN and LR predictions
• Add check for cube clock drift to polling programs, or find way to synchronize cube clocks
  • analyze logs to measure the clock drift
• Improved station and page annotations (Wiki annotations!) (3)
• Convert programs that currently use Location: to use CBI::WebRedirect (3)
• Should RTAS use RSS? (5)
• Fix tabulations so that bad values don't bleed through. (5, duff)
• put graphical benchmarks/datums/well on web (4)
• review QCHowTo documentation (4)
• Program to create benchmark descriptions in the PDB database from NGS database, and vice-versa (4, check with rizzo)
• QC more things (phone systems at Freeport, COE, CCRTNS, etc.) (4)
• More QC tools (view raw pressure data and power spectrum for wave data) (4)
• spanish voiceinfo (6)