[INSTRM-29] Define development archive storage operation and workflow Created: 07/Dec/16  Updated: 18/Mar/23  Resolved: 18/Mar/23

Status: Done
Project: Instrument control development
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Epic Priority: Major
Reporter: shimono Assignee: yuki.moritani
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Epic Name: devarch

 Description   

note: This is epic for initiate development archive storage service. File ticket for each individual task or issue.

  • fmadec develop possible (or desirable) way to push FITS files from LAM local storage
  • cloomis develop possible way to push FITS files from JHU test facility
  • define file distribution procedure, e.g. sync all files by periodical rsync, on-demand download by URI


 Comments   
Comment by fmadec [ 19/May/17 ]

According to the update from Atsuhsi it seems that IPMU have some constrains on ssh access.

So that LAM propose that the synchronisation comes from IPMU or PU to LAM and not from LAM to external site.
This access will only be using ssh and will be restricted to ideally only to the server that will pull the data.

We should agree on the frequency of the sync (ideally that could be done once a week during week-end)

What do you think?

Comment by shimono [ 31/May/17 ]

Restriction from outside-IPMU to global network (by means which has non-RFC1918 addresses) in IPMU is somehow messy thing now and its coordination/operation could be changed in short time scale. Web access (80/443) is relatively less restricted than ssh (22), so one "reliable" possibility is webdav-rsync if we want to push from outside-IPMU. For getting from this archive, there would be bunch of ways from pick one file by one via web, etc.

Also if instrument development sites (like LAM, PU, ASIAA) are easily accessible from outside, ssh-rsync from IPMU archive server is one possibility. This will require some configuration at development sites, that ssh gateway is possible to reach local data storage at lab (or local archive?). How difficult depends on configurations at each site, so it is required to be checked site by site.

For frequency, I think once a week at weekend could work. For immediate data transfer from LAM to PU for short time analysis, we could do/plan transfer on demand as now we are doing.

Comment by arnaud.lefur [ 18/Mar/23 ]

yuki.moritani do you want to keep this open ? or shall we close it .

Comment by yuki.moritani [ 18/Mar/23 ]

Since LAM data have been archiving, I close this ticket. Archiving JHU (or other institute) data will be handled under a new ticket, if we decide to do so. 

Generated at Sat Feb 10 16:20:46 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.