Cookbook to use LLMs effectively for sciences, in various levels of expertise

Use of LLMs in NASA’s Science Mission Directorate

Motivation

From a scientific perspective, the motivation behind leveraging large language models (LLMs) in research workflows is rooted in efficiency and the acceleration of discovery. Using these advanced models promises to significantly save scientists' time by streamlining research processes such as data and information discovery, access, literature reviews, and the writing or modification of code. Highlighting this point, Gentemann et al. (2021) underscored that a staggering eighty percent of a project's time is often consumed by "data-wrangling," leaving only twenty percent for analyzing results and disseminating learned insights. By integrating LLMs into the research workflow, we pave the way for moving science toward future discoveries at a much faster pace. Similarly, from a data system perspective, adopting LLMs enhances the visibility, accessibility, usability, and overall value of open data and information. This technology fosters increased visibility and awareness of available data through novel discovery mechanisms and applications and enhances access and usage by making it easier to find and access information. Furthermore, LLMs open up new pathways for analysis by enabling the discovery and combination of previously scattered data across disparate locations. In today's digital age, where there is an expectation to find and contextualize information quickly, LLMs can empower data systems to develop applications that align with these expectations, potentially freeing up resources to offer other value-added services to the community.

This tutorial aims to embrace the integration of AI, particularly LLMs, within the Science Mission Directorate's purview, fostering a culture of responsible and ethical AI usage. This culture involves critically examining the limitations inherent to AI and LLMs and the proactive establishment and implementation of ethical guidelines and guardrails to mitigate potential risks.

The tutorial aims to catalyze knowledge sharing and collaboration among scientists and researchers within NASA SMD, enabling them to leverage LLMs more effectively.

To this end, we will create a community-curated training on AI and LLMs, covering techniques such as Retriever-Augmented Generation (RAG), prompt engineering, and others, illustrating how these methods can be applied in scientific research. The training material will delve into the nuances that distinguish between an Encoder model and a Decoder model, ensuring participants gain a well-rounded understanding of the technological landscape of AI in science.

Through this tutorial, we aim to empower participants with the knowledge and skills necessary to navigate the evolving realm of AI LLMs and responsibly drive forward scientific inquiry and innovation.

The landscape of LLMs is in a constant state of rapid evolution, underpinned by a diverse and evolving infrastructure tailored for task-specific applications of LLMs. This dynamic environment is characterized by the frequent release of new and improved models, leading to an anticipated reduction in existing limitations over time. As the field progresses, we can expect an evolution in the methodologies and best practices to use them, adapting to the changing capabilities of these advanced tools.

A current trend within the landscape is the predominance of proprietary superior models. However, there is a strong expectation that open-source models will eventually emerge capable of meeting the specific needs of the scientific community and aligning with NASA principles of Open Science policy, as outlined in SPD-41. This alignment emphasizes the importance of accessibility, transparency, and collaboration in scientific research, mirroring the core values of the Open Science movement.

We must also be infrastructure agnostic, embracing platforms such as Azure, Bedrock, Vertex, and IBM WatsonX. This stance ensures flexibility and inclusivity in utilizing LLMs, allowing for a broad exploration of AI capabilities across different computational environments. By remaining open to the variety of infrastructures available, we position ourselves to adapt to and leverage the best tools for advancing scientific research regardless of their proprietary or open-source status.

Using LLMs, it is essential to remember the guiding principles of ethics for responsible research and innovation. These principles encourage the utilization of open models, data, workflows, and code whenever feasible, fostering transparency and collaboration within the scientific community. This approach aligns with NASA's Open Science initiatives and strengthens the integrity and reproducibility of research findings.

A critical aspect of engaging with LLMs, particularly in formulating inquiries, is the clarity and precision of the questions posed. In the spirit of Albert Einstein, the essence of solving complex problems lies in asking the right question. Investing time to refine your question can dramatically enhance the relevance and accuracy of AI-generated solutions, highlighting the notion that poorly constructed queries may lead to misleading or irrelevant answers.

Also, adopting a skeptical yet open mindset toward the information and outputs generated by LLMs is crucial. Carl Sagan's advocacy for the "baloney detection kit" is a valuable reminder to exercise critical thinking and seek independent verification of facts. This approach emphasizes the principles of 'trust but verify' and cautions against blind acceptance of generated content. Maintaining this carefulness ensures that one does not "check their brain at the door" but engages actively in the evaluative process.

Viewing AI as a collaborative partner rather than a standalone solution emphasizes the importance of human oversight in the decision-making process. The analogy of a co-pilot aptly illustrates this relationship; just as a co-pilot supports the pilot in navigating and ensuring the aircraft's safety, LLMs can assist researchers in exploring information and data. The responsibility to verify the validity and accuracy of the LLM outputs rests with the user. This partnership highlights the need for continuous engagement and verification by researchers to ensure the integrity and reliability of the generated content, reinforcing the principle that one should never let the AI "fly the ship unattended."

Get your credentials and other information using https://creds-workshop.nasa-impact.net/

Navigate to https://nasa-impact.awsapps.com/start#/

Log in using the credential provided

Navigate to the Applications tab

Click and open Amazon SageMaker Studio

Once the Studio starts, Click on JupyterLab

Click Create JupyterLab Space

Give it a name. Eg: Workshop

Once initialized, change Instance type to ml.t3.large and storage to 50

Click on Run Space. If it throws an error, you might have to pick an Image. The top setting called Latest works.

Open the Jupyter lab instance.

Clone this repository git clone https://github.com/nasa-impact/smd-llm-workshop.git

a. Click git b. Click on Git Clone Repo c. Paste https://github.com/nasa-impact/smd-llm-workshop.git and Click on Clone.

Prompt Engineering for Science

Introduction

Fundamental LLM Definitions

Tokens

The basic unit of input a LLM uses. It can be a word, part of a word, or some other segment of input. Text is converted to numbers by a tokenizer. Tokens are then used to

A mathematical representation of words that represent meaning and context. An embedding is essentially a list of numbers for all the dimensions the embedding represents. This list is also called a vector.

There are multiple types of LLM parameters. The types of parameters of interest to this group include:

A prompt is the instructions provided to an LLM when starting an interaction or making a request. Prompts can be simple questions or a larger set of instructions.

Tasks undertaken to potentially improve the behavior of the LLM. Fine-tuning is typically undertaken to overcome prompt engineering limitations.

Reference: Gartner: What Technical Professionals Need to Know About Large Language Models. By Analyst Wilco van Ginkel.

Prompt engineering is the design and optimization of effective queries or instructions (aka prompt patterns) when using generative AI tools like ChatGPT in order to get desired responses. Understanding how to write prompts will increase the quality of responses received. In other words, ask the right questions to get better answers.

There is no perfect LLM - instead there are LLMs that are appropriate for your use case and your needs. We’ve provided a quick guide of the most commonly used models here for your reference.

Available prompt parameters may vary depending on the interface you are working in. Note that the model that you select may determine the parameters available. Here are some typical parameters you can control along with some definitions:

Reference: Pluralsight course - Getting Started on Prompt Engineering with Generative AI by Amber Israelsen.

When creating an initial prompt, begin the prompt with an action verb (Adopt, Create, Summarize, etc…).

Iteration on the prompt may be required until the output meets your expectations.

Avoid long sessions. Restart sessions when you need to reset context or want to provide different prompting instructions.

Do not share sensitive information.

Privacy: Avoid requesting or including personal information; adhere to privacy and security protocols.

Like a human, attention can be a problem with larger portions of text or conversations. To manage attention, it helps to break content into smaller sections.

Misinformation: Reference scientific consensus from credible, peer-reviewed sources. When using RAG patterns, leverage authoritative sources like the Science Discovery Engine index.

Bias/Fairness: Utilize RAG patterns with authoritative, curated sources for balanced coverage. Verify fairness in citations.

Ownership/Copyright: Generate original content, respecting copyright laws. When using RAG patterns, leverage authoritative sources like the Science Discovery Engine index and provide proper citations.

Transparency: Ask the model to explain the reasoning behind AI outputs or use prompts like ‘fact check’ to help verify. When using RAG patterns, leverage authoritative sources like the Science Discovery Engine index and provide proper citations.

Prompt Patterns for Science

Let's explore some common prompt patterns as templates that can be re-used for various scientific tasks. These patterns are designed to be used as a starting point for prompt engineering, and can be modified to fit the specific needs of the task at hand. To quickly get started in using the patterns, replace the [placeholders] with the relevant information for your task.

Reference: White et al. ‘A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT.’ https://arxiv.org/abs/2302.11382

These prompts focus on tailoring or guiding the format, structure or type of output provided by the LLM.

This pattern provides limits to ultimately output a sequence of steps given some partially

provided “ingredients” that must be provided in a series of steps to achieve a stated goal. This prompt is helpful for tasks when the user knows the desired end result and the ingredients needed to achieve the result but not the detailed steps themselves.

“I am trying to [complete a task]. I know that I need [step A, B, C]. Please provide a complete sequence of steps. Please fill in any missing steps.”

I am trying to preprocess Landsat 8 Level-1 data. I know that I need to find and download the data. I know that I need to complete georeferencing, conversion to radiance, solar corrections and atmospheric corrections. I know I will use the ENVI software.