Digital Museum of Web Content for Future Generations
Some websites may shut down or disappear for numerous reasons. What if you wanted to look at those websites again? So how do you find this historical data without loss?
Irrespective of the field of work, one cannot deny the importance of maintaining vital information such as business records, government documents, campaign materials, valuable research, important videos and interviews. For that matter everything on the web itself is a record and it’s necessary to preserve and present to the future generations.
The average life span of a web page is about 100 days and approximately 10% of the information on social media sites is lost even before the first year of its creation. The content we share on social media may exist forever but its life span is relatively shorter.
Website Archiving is the most hidden truth on the web, let’s have a look at what it is all about and how important it is for any business or technology enthusiast who dotes to dig the past for the future predictions.
What is web archiving?
Web archiving is the process of preserving a duplicate of the website offline as a facsimile which can be available even if the replicated website is subjected to change or completely taken down with time. It is archived in the history of a topic that is available as a PDF document or a media file. These historical records of affluent information sites can be captured on the regular basis and integrated to the archives so that the future generations will have a robust record of research information, events, and projects.
Why Should We
Over the last couple of decades, much of our lives have moved online. Ever since the beginning of human evolution, information about ages and times is stored on various channels from stone carvings to historical books to learn from the past and assess the present and predict the future. With everything being digital these days where will the historians of all time look?
The large part of the web is not lost forever. We have Web Archives.
Web archives contain copies of pages from the live web along with the information and the record of the date on which they have archived. National institutions and digital libraries have been busy collecting this online information since 1990’s and there are now available to everyone who wants to search the past.
WHAT YOU CAN FIND THERE?
As the internet kept evolving for the past 20 years, so is the archiving process. In the beginning, only the pay layout and images used to be stored but with time things got better and web crawling can now capture most of the website features and information before it gets disappeared. Today every news and media channel archive couple of times a day before they edit or delete something valuable.
But web archiving has its own limitation as it cannot capture everything. Most of our online presence i.e., Intranet information, Emails, and social media presence cannot be captured due to technicalities and API integrations that inhibit any web crawlers from capturing the information. Even with these setbacks, there is a huge amount of that terabytes of data and trillions of words and millions of memories available.
FUTURE OF WEB ARCHIVING
The internet has enabled many platforms of knowledge sharing, innovation by connecting people around the world. It also set many challenges for the institutions to document and preserve this enormous amount of data.
Data science, Artificial Intelligence, and Machine learning are the buzz words of tech innovations these days. Algorithms are the core of this digital disruption but how far we can rely on these algorithms to preserve the history of times is the question of the hour.
It’s an undeniable truth that the future is going to be dictated by these so-called algorithms and machine learning algorithms. In simple words, using these algorithms to think and act like humans. To facilitate the same these algorithms need a huge amount of datasets to understand and learn.
But the problem is with the “data” that is getting archived for these cutting-edge technology projects. Most of this data is biased even after cleaning the data without outliers. Thus the quality of data is going to dictate the behavioral characteristics of the product.
Case studies of web archiving and its future
LINK ANALYSIS
A large number of websites are captured and the links and data are mined to analyze the relationships between the content and ideas over time. The web archive datasets are used for this purpose to analyze the change over time on the subjected content.
A similar kind of link analysis study of approximately 1.3 billion URLs crawled by Common Crawl in 2012 gave the following key findings:
- 22% of web pages contain Facebook URLs
- Among 500M hardcoded links to Facebook, only 3.5 million are unique.
This study has provided an accurate data on how Facebook is influencing the web. And the same type of link analysis is still in practice to study social platforms and compare the relative trends of the web.
ACCOUNTABILITY AND LEGAL ISSUES
In the court of law where evidence place an important role, cybercrimes are chased using the archives of digital profiles and transactions on websites are playing an important role in the matter of accountability. Often when dealing with cases like patent infringements, hacking of the websites and modifying the digital born content and copyright infringements the court of law is considering the archives as a considerable proof of evidence. Thus archiving is going to play role in the court of law in the digital age to resolve the legal issues.
FINANCIAL SERVICES
Ever since the emergence of social media, billions of users are going online for every small need. Digital transactions and especially in the countries where payments are mostly cashless the ability to archive the transactions of the users and business is an undeniable thing for any financial firm to put into practice right now so that they can keep track of business and act accordingly.
DIGITALISING NEWS AND PUBLICATIONS
When we are talking about the volatile nature of information on the web, one such thing that changes every day is the newspaper content. Digitising the born-digital material like news in archives is an excellent way of making those accessible to the target audience and preserving the history for the future generations.
TEXT MINING
Text Mining is the process of using the archive websites for the research studies to extract, visualize and analyze the speech and text usage of the crawled websites. This kind of text mining is used to determine the emotions used when discussing certain topics. Such kind of analysis and outputs are of a great importance to input for the speech recognition systems and voice assistants like Alexa to be more authentic in response to human queries.
ANALYSIS OF TECHNOLOGY TRENDS
The captured websites archive collections show the timeline development of web and its web technologies. This timeline keeps a track of the changes in programming languages and their usage over time, highlighting the trends of technologies in use. For example evolution of JavaScript over time on the web
Listed above are only few studies to explain the importance of web archiving but the archiving is all about selecting the data that matters, harvesting it via software to reproduce the same at every instant, preserving it subject to the rules and best policies so that it can access by the future coworkers, researchers, historians and public.