As it is already known, modern observatories like the Atacama Large Millimeter/submillimeter Array (ALMA) and the Very Long Baseline Array (VLBA) generate large-scale data, which will be accentuated with the incorporation of new observatories, such as the Square Kilometre Array (SKA). It is projected by 2020 to obtain an archived astronomical data in a PB-scale (60 Petabyte). The Chilean Virtual Observatory (ChiVO) has stored the spectral cubes of ALMA and seeks to offer these data openly to the community, but downloading and processing these data should be done in its facilities. To this end, our proposal considers the cubes as a high order tensor, specifically 3-way tensor with 2 spatial dimensions (galactic latitude and longitude), and a velocity dimension. This opens a new approach and opportunity for computational prohibitive massive analysis of these cubes. Based on this premise, we propose TensorFit, a natural and scalable library to handle spectral cubes in a tensor mode. The implementation is built on parallel oriented frameworks, and distributed processing of n-arrays on PyTorch (GPU and CPU). To verify the impact of this proposal, our focus is on showing the benefits of tensor compression, in particular to Tucker implementations. These have demonstrated outstanding results of dimensionality reduction of multidimensional data in other scientific domains.

Performing astronomical data analysis using only personal computers is becoming impractical for the very large data sets produced nowadays. As analysis is not a task that can be automatized to its full extent, the idea of moving processing where the data is located means also moving the whole scientific process towards the archives and data centers. Using Jupyter Notebooks as a remote service is a recent trend in data analysis that aims to deal with this problem, but harnessing the infrastructure to serve the astronomer without increasing the complexity of the service is a challenge. In this paper we present the architecture and features of JOVIAL, a Cloud service where astronomers can safely use Jupyter notebooks over a personal space designed for high-performance processing under the high-availability principle. We show that features existing only in specific packages can be adapted to run in the notebooks, and that algorithms can be adapted to run across the data center without necessarily redesigning them.

The big data problem in Astronomy is a well now know issue, but in majority of the cases is constrains to volume of this data. Our propose take care of another aspect of the problem: the dimensionality problem, in the scope of multidimensional data especially Astronomical data cubes. We use tensor decompositions for two goals, first using tucker we achieve super compression rates that allowed to saving until 91% disks space and network traffic and second using a CANDECOMP/PARAFAC or Canonical Decomposition (CP) we build a system to find the multi-linear manifold in this astronomical cubes. Because this is a problem of BigData, for ours library we test three implementations: One approach over an intense use of GPU supported by PyTorch and using the traditional approach of HPC using MPI. Our proposed start from a simple but powerful idea, if we are dealing with multidimensional data (astronomical cubes), Why are we limited to use bi-dimensional techniques?. For example we use PCA for dimensionality reductions in spectral cubes instead of multidimensional approach that preserve the multi linear manifold inside this multidimensional data, we propose to pass from a linear algebra approach to a multi-linear algebra approach using tensor theory.

The research on computational methods for astronomy performed by the first phase of the Chilean Virtual Observatory (ChiVO) led to the development of functional prototypes, implementing state-of-the-art computational methods and proposing new algorithms and techniques. The ChiVO software architecture is based on the use of the IVOA protocols and standards. These protocols and standards are grouped in layers, with emphasis on the application and data layers, because their basic standards define the minimum operation that a VO should conduct. As momentary verification, the current implementation works with a set of data, with 1 TB capacity, which comes from the reduction of the cycle 0 of ALMA. This research was mainly focused on spectroscopic data cubes coming from the cycle 0 ALMA's public data. As the dataset size increases when the cycle 1 ALMA's public data is also increasing every month, data processing is becoming a major bottleneck for scientific research in astronomy. When designing the ChiVO, we focused on improving both computation and I/ O costs, and this led us to configure a data center with 424 high speed cores of 2,6 GHz, 1 PB of storage (distributed in hard disk drives-HDD and solid state drive-SSD) and high speed communication Infiniband. We are developing a cloud based e-infrastructure for ChiVO services, in order to have a coherent framework for developing novel web services for on-line data processing in the ChiVO. We are currently parallelizing these new algorithms and techniques using HPC tools to speed up big data processing, and we will report our results in terms of data size, data distribution, number of cores and response time, in order to compare different processing and storage configurations. © (2016) COPYRIGHT Society of Photo-Optical Instrumentation Engineers (SPIE). Downloading of the abstract is permitted for personal use only.

Methods for observing space have changed profoundly in the past few decades. The methods needed to detect and record astronomical objects have shifted from conventional observations in the optical range to more sophisticated methods which permit the detection of not only the shape of an object but also the velocity and frequency of emissions in the millimeter-scale wavelength range and the chemical substances from which they originate. The consolidation of radio astronomy through a range of global-scale projects such as the Very Long Baseline Array (VLBA) and the Atacama Large Millimeter/submillimeter Array (ALMA) reinforces the need to develop better methods of data processing that can automatically detect regions of interest (ROIs) within data cubes (position–position–velocity), index them and facilitate subsequent searches via methods based on queries using spatial coordinates and/or velocity ranges. In this article, we present the development of an automatic system for indexing ROIs in data cubes that is capable of automatically detecting and recording ROIs while reducing the necessary storage space. The system is able to process data cubes containing megabytes of data in fractions of a second without human supervision, thus allowing it to be incorporated into a production line for displaying objects in a virtual observatory. We conducted a set of comprehensive experiments to illustrate how our system works. As a result, an index of 3% of the input size was stored in a spatial database, representing a compression ratio equal to 33:1 over an input of 20.875 GB, achieving an index of 773 MB approximately. On the other hand, a single query can be evaluated over our system in a fraction of second, showing that the indexing step works as a shock-absorber of the computational time involved in data cube processing. The system forms part of the Chilean Virtual Observatory (ChiVO), an initiative which belongs to the International Virtual Observatory Alliance (IVOA) that seeks to provide the capability of content-based searches on data cubes to the astronomical community.

This article shows that high costs to obtain data in astronomy justifies the use of cloud computing. In such a field, like others in science, there are several benefits when this data is freely available to the whole scientific community. The availability of these data will foster innovative applications to exploit and explore data. These applications could be offered as a service in the cloud. A National Data Center for e-Science could take advantage of these high costs data and offer open data to the community to maximize data accessibility.

This paper presents the challenges, architecture and current status of the Chilean Virtual Observatory (ChiVO), which is a software infrastructure for accessing and processing astronomical data generated in Chile. As ChiVO is part of the International Virtual Observatory Alliance (IVOA), we strictly follow the protocols and standards that this organization produce. However, there are always open challenges due to the new observational technologies and local requirements that motivates research on every new virtual observatory, such as the complex data models and Big Data problems that the ALMA Observatory is confronting. The current ChiVO prototype includes IVOA compliant services as well as new solutions designed for ALMA data, all of them using modern software technologies.

This paper presents a short survey on the astronomical Virtual Observatories (VOs) that are part of the International Virtual Observatory Alliance (IVOA). From how they are distributed worldwide to their specialization range on the electromagnetic spectrum, we summarize key aspects of the current 21 VO members. Through the registry service, we explore the resources that the VO offer as a whole, and identify that even though the VO is already a mature initiative, there are important challenges to address in the near future.

The size and quantity of the data that is being generated by large astronomical projects like ALMA, requires a paradigm change in astronomical data analysis. Complex data, such as highly sensitive spectroscopic data in the form of large data cubes, are not only difficult to manage, transfer and visualize, but they also turn unfeasible the use of traditional data analysis techniques and algorithms. Consequently, the attention have been placed on machine learning and artificial intelligence techniques, to develop approximate and adaptive methods for astronomical data analysis within a reasonable computational time. Unfortunately, these techniques are usually sub optimal, stochastic and strongly dependent of the parameters, which could easily turn into "a ghost in the machine" for astronomers and practitioners. Therefore, a proper assessment of these methods is not only desirable but mandatory for trusting them in large-scale usage. The problem is that positively verifiable results are scarce in astronomy, and moreover, science using bleeding-edge instrumentation naturally lacks of reference values. We propose an Astronomical SYnthetic Data Observatory (ASYDO), a virtual service that generates synthetic spectroscopic data in the form of data cubes. The objective of the tool is not to produce accurate astrophysical simulations, but to generate a large number of labelled synthetic data, to assess advanced computing algorithms for astronomy and to develop novel Big Data algorithms. The synthetic data is generated using a set of spectral lines, template functions for spatial and spectral distributions, and simple models that produce reasonable synthetic observations. Emission lines are obtained automatically using IVOA's SLAP protocol (or from a relational database) and their spectral profiles correspond to distributions in the exponential family. The spatial distributions correspond to simple functions (e.g., 2D Gaussian), or to scalable template objects. The intensity, broadening and radial velocity of each line is given by very simple and naive physical models, yet ASYDO's generic implementation supports new user-made models, which potentially allows adding more realistic simulations. The resulting data cube is saved as a FITS file, also including all the tables and images used for generating the cube. We expect to implement ASYDO as a virtual observatory service in the near future.

Currently, the standards and protocols for data access in the Virtual Observatory architecture (DAL) are generally implemented with relational databases based on SQL. In particular, the Astronomical Data Query Language (ADQL), language used by IVOA to represent queries to VO services, was created to satisfy the different data access protocols, such as Simple Cone Search. ADQL is based in SQL92, and has extra functionality implemented using PgSphere. An emergent alternative to SQL are the so called NoSQL databases, which can be classified in several categories such as Column, Document, Key-Value, Graph, Object, etc.; each one recommended for different scenarios. Within their notable characteristics we can find: schema-free, easy replication support, simple API, Big Data, etc. The Chilean Virtual Observatory (ChiVO) is developing a functional prototype based on the IVOA architecture, with the following relevant factors: Performance, Scalability, Flexibility, Complexity, and Functionality. Currently, it's very difficult to compare these factors, due to a lack of alternatives. The objective of this paper is to compare NoSQL alternatives with SQL through the implementation of a Web API REST that satisfies ChiVO's needs: a SESAME-style name resolver for the data from ALMA. Therefore, we propose a test scenario by configuring a NoSQL database with data from different sources and evaluating the feasibility of creating a Simple Cone Search service and its performance. This comparison will allow to pave the way for the application of Big Data databases in the Virtual Observatory.

The study of the astronomical structures is important to the astronomical community because it can help to identify objects, which can be classified based on their internal structure or their relation to other objects. For this reason, it is developed an automated tool to analyze astronomical images into its components. Firstly, a 2D images is decomposed into different spatial scales based on wavelet transform. Then, it is implemented a detection algorithms to each spatial scale, such as Clumpfind, Gaussclump, or Dendrogram techniques. The goal is to build a new algorithm and tool that is available to the community and satisfies the requirements of the next Chilean Virtual Observatory (ChiVO).

The Virtual Observatories strive to interoperate, exchange data and share services as if it was only one big VO. In this work, the state of the art of VOs will be presented and summarized in a schematic diagram with the frequency range of the observed data that every VO publishes. Chile, currently a member of the IVOA, collaborates with the Atacama Large Millimeter/submillimeter Array (ALMA), to study and propose ways to adequate the data generated by ALMA to the different data model proposed by the IVOA.

The success of an observatory is usually measured by its impact in the scientific community, so a common objective is to provide transparent ways to access the generated data. The Chilean Virtual Observatory (ChiVO), started working in the implementation of a prototype, in collaboration with ALMA, considering the current needs of the Chilean astronomical community, in addition to the protocols and standards of IVOA, and the comparison of different existing data access toolkit services. Based on this efforts, a VO prototype was designed and implemented for the ALMA large scale of data.