From e12dc5b0e85604fca711f19d98e6bf2040e83885 Mon Sep 17 00:00:00 2001 From: Randall Degges Date: Tue, 20 Aug 2024 20:51:23 -0700 Subject: [PATCH 01/60] Adding new pattern: create_recursive_outline. This pattern is actually based on this incredibly great article: https://learnhowtolearn.org/how-to-build-extremely-quickly/ The idea is to use this pattern whenever you want to break an idea or task down into small components, fully fleshing out your own TODO list of things to implement to get it working. This applies to things like writing articles/papers, creating applications, and much more. --- patterns/create_recursive_outline/system.md | 59 +++++++++++++++++++++ 1 file changed, 59 insertions(+) create mode 100644 patterns/create_recursive_outline/system.md diff --git a/patterns/create_recursive_outline/system.md b/patterns/create_recursive_outline/system.md new file mode 100644 index 0000000..dcbe2e2 --- /dev/null +++ b/patterns/create_recursive_outline/system.md @@ -0,0 +1,59 @@ +# IDENTITY and PURPOSE + +You are an AI assistant specialized in task decomposition and recursive outlining. Your primary role is to take complex tasks, projects, or ideas and break them down into smaller, more manageable components. You excel at identifying the core purpose of any given task and systematically creating hierarchical outlines that capture all essential elements. Your expertise lies in recursively analyzing each component, ensuring that every aspect is broken down to its simplest, actionable form. + +Whether it's an article that needs structuring or an application that requires development planning, you approach each task with the same methodical precision. You are adept at recognizing when a subtask has reached a level of simplicity that requires no further breakdown, ensuring that the final outline is comprehensive yet practical. + +Take a step back and think step-by-step about how to achieve the best possible results by following the steps below. + +# STEPS + +- Identify the main task or project presented by the user + +- Determine the overall purpose or goal of the task + +- Create a high-level outline of the main components or sections needed to complete the task + +- For each main component or section: + - Identify its specific purpose + - Break it down into smaller subtasks or subsections + - Continue this process recursively until each subtask is simple enough to not require further breakdown + +- Review the entire outline to ensure completeness and logical flow + +- Present the finalized recursive outline to the user + +# OUTPUT INSTRUCTIONS + +- Only output Markdown + +- Use hierarchical bullet points to represent the recursive nature of the outline + +- Main components should be represented by top-level bullets + +- Subtasks should be indented under their parent tasks + +- If subtasks need to be broken down as well, they should be indented under their parent tasks + +- Include brief explanations or clarifications for each component or task where necessary + +- Use formatting (bold, italic) to highlight key points or task categories + +- If the task is an article: + - Include a brief introduction stating the article's purpose + - Outline main sections with subsections + - Break down each section into key points or paragraphs + +- If the task is an application: + - Include a brief description of the application's purpose + - Outline main components (e.g., frontend, backend, database) + - Break down each component into specific features or development tasks + - Include specific implementation information as necessary (e.g., one sub-task might read "Store user-uploaded files in an object store" + +- Ensure that the lowest level tasks are simple and actionable, requiring no further explanation + +- Ensure you follow ALL these instructions when creating your output + +# INPUT + +INPUT: From 151fff8f8dbf8f66edc57a1151497504529d9792 Mon Sep 17 00:00:00 2001 From: James Craigen Date: Wed, 21 Aug 2024 19:03:55 +0100 Subject: [PATCH 02/60] Fix broken link in table of contents in README.md (Migrating -> Migration) --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index eddcf16..9011521 100644 --- a/README.md +++ b/README.md @@ -34,7 +34,7 @@ - [Too many prompts](#too-many-prompts) - [The Fabric approach to prompting](#our-approach-to-prompting) - [Installation](#Installation) - - [Migrating](#Migrating) + - [Migration](#Migration) - [Upgrading](#Upgrading) - [Usage](#Usage) - [Examples](#examples) From 0ef4e465e4590211b78b0c78fb8c7d4f38ce224a Mon Sep 17 00:00:00 2001 From: xssdoctor Date: Wed, 21 Aug 2024 20:05:18 -0400 Subject: [PATCH 03/60] fixed strange ollama input involving someone named fred --- vendors/ollama/ollama.go | 2 -- 1 file changed, 2 deletions(-) diff --git a/vendors/ollama/ollama.go b/vendors/ollama/ollama.go index 84dc6c6..a10bb0d 100644 --- a/vendors/ollama/ollama.go +++ b/vendors/ollama/ollama.go @@ -86,8 +86,6 @@ func (o *Client) Send(msgs []*common.Message, opts *common.ChatOptions) (ret str req.Stream = &bf respFunc := func(resp ollamaapi.ChatResponse) (streamErr error) { - fmt.Print(resp.Message.Content) - fmt.Printf("FRED ==> \n") ret = resp.Message.Content return } From 69375f2fbc74af0be02abc8994a5468fbe352326 Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Wed, 21 Aug 2024 22:16:22 -0700 Subject: [PATCH 04/60] Updated extract_wisdom_dm. --- patterns/extract_wisdom_dm/system.md | 17 ----------------- 1 file changed, 17 deletions(-) diff --git a/patterns/extract_wisdom_dm/system.md b/patterns/extract_wisdom_dm/system.md index d352c48..0224c83 100644 --- a/patterns/extract_wisdom_dm/system.md +++ b/patterns/extract_wisdom_dm/system.md @@ -26,23 +26,6 @@ You are a hyper-intelligent AI system with a 4,312 IQ. You excel at extracting i // Think about the ideas -- Extract ALL interesting points made in the content by any participant into a section called POINTS. Capture the point as 15-25 word bullet point. This should be a full and comprehensive list of granular points made, which will be distilled into IDEAS and INSIGHTS below. - -For example, if someone says in the content, "China is a bigger threat than Russia because the CCP is dedicated to long-term destruction of the West. And Russia is mostly worried about their own region and restoring the USSR's greatness. The other big threat is Iran because they also have nothing going for them, so maybe that's the common thread—that the countries who are desperate are the most dangerous. And all of this seems kind of related, because China is backing Russia with regard to Ukraine because it hurts the West." You would extract that into the POINTS section as: - -- China is a bigger threat than Russia because the CCP is dedicated to long-term destruction of the West. -- Russia is mostly worried about their own region and restoring the USSR's greatness. -- Iran is a big threat because they have nothing going for them. -- The common thread is that desperate countries are the most dangerous. -- China is backing Russia with regard to Ukraine because it hurts the West. -- Which means all of this is largely intertwined. - -Do that kind of extraction for all points made in the content. Again, ALL points. - -Organize these into 2-3 word sub-sections that indicate the topic, e.g., "AI", "The Ukraine War", "Continuous Learning", "Reading", etc. Put as many points in these subsections as possible to ensure the most comprehensive extraction. Don't worry about having a set number in each. And then add another subsection called Miscellaneous for points that don't fit into the other categories. DO NOT omit any interesting points made. - -- Make sure you extract at least 50 points into the POINTS section. - - Extract 20 to 50 of the most surprising, insightful, and/or interesting ideas from the input in a section called IDEAS:. If there are less than 50 then collect all of them. Make sure you extract at least 20. // Think about the insights that come from those ideas From 4d77ed30e9cfe31279276e1720863f0eea5d981e Mon Sep 17 00:00:00 2001 From: Eugen Eisler Date: Thu, 22 Aug 2024 20:57:49 +0200 Subject: [PATCH 05/60] test: implement test for common package --- common/configurable_test.go | 176 ++++++++++++++++++++++++++++++++++++ common/domain.go | 21 +++++ common/domain_test.go | 25 +++++ common/messages.go | 22 ----- common/vendor.go | 12 --- core/chatter.go | 3 +- core/vendors.go | 14 +-- go.mod | 4 + utils/log.go | 28 ------ vendors/vendor.go | 14 +++ 10 files changed, 249 insertions(+), 70 deletions(-) create mode 100644 common/configurable_test.go create mode 100644 common/domain_test.go delete mode 100644 common/messages.go delete mode 100644 common/vendor.go delete mode 100644 utils/log.go create mode 100644 vendors/vendor.go diff --git a/common/configurable_test.go b/common/configurable_test.go new file mode 100644 index 0000000..4c212db --- /dev/null +++ b/common/configurable_test.go @@ -0,0 +1,176 @@ +package common + +import ( + "bytes" + "os" + "testing" + + "github.com/stretchr/testify/assert" +) + +func TestConfigurable_AddSetting(t *testing.T) { + conf := &Configurable{ + Settings: Settings{}, + Label: "TestConfigurable", + EnvNamePrefix: "TEST_", + } + + setting := conf.AddSetting("test_setting", true) + assert.Equal(t, "TEST_test_setting", setting.EnvVariable) + assert.True(t, setting.Required) + assert.Contains(t, conf.Settings, setting) +} + +func TestConfigurable_Configure(t *testing.T) { + setting := &Setting{ + EnvVariable: "TEST_SETTING", + Required: true, + } + conf := &Configurable{ + Settings: Settings{setting}, + Label: "TestConfigurable", + } + + os.Setenv("TEST_SETTING", "test_value") + err := conf.Configure() + assert.NoError(t, err) + assert.Equal(t, "test_value", setting.Value) +} + +func TestConfigurable_Setup(t *testing.T) { + setting := &Setting{ + EnvVariable: "TEST_SETTING", + Required: false, + } + conf := &Configurable{ + Settings: Settings{setting}, + Label: "TestConfigurable", + } + + err := conf.Setup() + assert.NoError(t, err) +} + +func TestSetting_IsValid(t *testing.T) { + setting := &Setting{ + EnvVariable: "TEST_SETTING", + Value: "some_value", + Required: true, + } + + assert.True(t, setting.IsValid()) + + setting.Value = "" + assert.False(t, setting.IsValid()) +} + +func TestSetting_Configure(t *testing.T) { + os.Setenv("TEST_SETTING", "test_value") + setting := &Setting{ + EnvVariable: "TEST_SETTING", + Required: true, + } + err := setting.Configure() + assert.NoError(t, err) + assert.Equal(t, "test_value", setting.Value) +} + +func TestSetting_FillEnvFileContent(t *testing.T) { + buffer := &bytes.Buffer{} + setting := &Setting{ + EnvVariable: "TEST_SETTING", + Value: "test_value", + } + setting.FillEnvFileContent(buffer) + + expected := "TEST_SETTING=test_value\n" + assert.Equal(t, expected, buffer.String()) +} + +func TestSetting_Print(t *testing.T) { + setting := &Setting{ + EnvVariable: "TEST_SETTING", + Value: "test_value", + } + expected := "TEST_SETTING: test_value\n" + fmtOutput := captureOutput(func() { + setting.Print() + }) + assert.Equal(t, expected, fmtOutput) +} + +func TestSetupQuestion_Ask(t *testing.T) { + setting := &Setting{ + EnvVariable: "TEST_SETTING", + Required: true, + } + question := &SetupQuestion{ + Setting: setting, + Question: "Enter test setting:", + } + input := "user_value\n" + fmtInput := captureInput(input) + defer fmtInput() + err := question.Ask("TestConfigurable") + assert.NoError(t, err) + assert.Equal(t, "user_value", setting.Value) +} + +func TestSettings_IsConfigured(t *testing.T) { + settings := Settings{ + {EnvVariable: "TEST_SETTING1", Value: "value1", Required: true}, + {EnvVariable: "TEST_SETTING2", Value: "", Required: false}, + } + + assert.True(t, settings.IsConfigured()) + + settings[0].Value = "" + assert.False(t, settings.IsConfigured()) +} + +func TestSettings_Configure(t *testing.T) { + os.Setenv("TEST_SETTING", "test_value") + settings := Settings{ + {EnvVariable: "TEST_SETTING", Required: true}, + } + + err := settings.Configure() + assert.NoError(t, err) + assert.Equal(t, "test_value", settings[0].Value) +} + +func TestSettings_FillEnvFileContent(t *testing.T) { + buffer := &bytes.Buffer{} + settings := Settings{ + {EnvVariable: "TEST_SETTING", Value: "test_value"}, + } + settings.FillEnvFileContent(buffer) + + expected := "TEST_SETTING=test_value\n" + assert.Equal(t, expected, buffer.String()) +} + +// captureOutput captures the output of a function call +func captureOutput(f func()) string { + var buf bytes.Buffer + stdout := os.Stdout + r, w, _ := os.Pipe() + os.Stdout = w + f() + _ = w.Close() + os.Stdout = stdout + buf.ReadFrom(r) + return buf.String() +} + +// captureInput captures the input for a function call +func captureInput(input string) func() { + r, w, _ := os.Pipe() + _, _ = w.WriteString(input) + w.Close() + stdin := os.Stdin + os.Stdin = r + return func() { + os.Stdin = stdin + } +} diff --git a/common/domain.go b/common/domain.go index b11458c..f5a1e40 100644 --- a/common/domain.go +++ b/common/domain.go @@ -19,3 +19,24 @@ type ChatOptions struct { PresencePenalty float64 FrequencyPenalty float64 } + +// NormalizeMessages remove empty messages and ensure messages order user-assist-user +func NormalizeMessages(msgs []*Message, defaultUserMessage string) (ret []*Message) { + // Iterate over messages to enforce the odd position rule for user messages + fullMessageIndex := 0 + for _, message := range msgs { + if message.Content == "" { + // Skip empty messages as the anthropic API doesn't accept them + continue + } + + // Ensure, that each odd position shall be a user message + if fullMessageIndex%2 == 0 && message.Role != "user" { + ret = append(ret, &Message{Role: "user", Content: defaultUserMessage}) + fullMessageIndex++ + } + ret = append(ret, message) + fullMessageIndex++ + } + return +} diff --git a/common/domain_test.go b/common/domain_test.go new file mode 100644 index 0000000..a4b5ffe --- /dev/null +++ b/common/domain_test.go @@ -0,0 +1,25 @@ +package common + +import ( + "github.com/stretchr/testify/assert" + "testing" +) + +func TestNormalizeMessages(t *testing.T) { + msgs := []*Message{ + {Role: "user", Content: "Hello"}, + {Role: "bot", Content: "Hi there!"}, + {Role: "bot", Content: ""}, + {Role: "user", Content: ""}, + {Role: "user", Content: "How are you?"}, + } + + expected := []*Message{ + {Role: "user", Content: "Hello"}, + {Role: "bot", Content: "Hi there!"}, + {Role: "user", Content: "How are you?"}, + } + + actual := NormalizeMessages(msgs, "default") + assert.Equal(t, expected, actual) +} diff --git a/common/messages.go b/common/messages.go deleted file mode 100644 index dd1e330..0000000 --- a/common/messages.go +++ /dev/null @@ -1,22 +0,0 @@ -package common - -// NormalizeMessages remove empty messages and ensure messages order user-assist-user -func NormalizeMessages(msgs []*Message, defaultUserMessage string) (ret []*Message) { - // Iterate over messages to enforce the odd position rule for user messages - fullMessageIndex := 0 - for _, message := range msgs { - if message.Content == "" { - // Skip empty messages as the anthropic API doesn't accept them - continue - } - - // Ensure, that each odd position shall be a user message - if fullMessageIndex%2 == 0 && message.Role != "user" { - ret = append(ret, &Message{Role: "user", Content: defaultUserMessage}) - fullMessageIndex++ - } - ret = append(ret, message) - fullMessageIndex++ - } - return -} diff --git a/common/vendor.go b/common/vendor.go deleted file mode 100644 index d5c7aa0..0000000 --- a/common/vendor.go +++ /dev/null @@ -1,12 +0,0 @@ -package common - -type Vendor interface { - GetName() string - IsConfigured() bool - Configure() error - ListModels() ([]string, error) - SendStream([]*Message, *ChatOptions, chan string) error - Send([]*Message, *ChatOptions) (string, error) - GetSettings() Settings - Setup() error -} diff --git a/core/chatter.go b/core/chatter.go index f9c5c7f..70123f3 100644 --- a/core/chatter.go +++ b/core/chatter.go @@ -4,6 +4,7 @@ import ( "fmt" "github.com/danielmiessler/fabric/common" "github.com/danielmiessler/fabric/db" + "github.com/danielmiessler/fabric/vendors" ) type Chatter struct { @@ -12,7 +13,7 @@ type Chatter struct { Stream bool model string - vendor common.Vendor + vendor vendors.Vendor } func (o *Chatter) Send(request *common.ChatRequest, opts *common.ChatOptions) (message string, err error) { diff --git a/core/vendors.go b/core/vendors.go index ec1629c..b81d26b 100644 --- a/core/vendors.go +++ b/core/vendors.go @@ -3,29 +3,29 @@ package core import ( "context" "fmt" - "github.com/danielmiessler/fabric/common" + "github.com/danielmiessler/fabric/vendors" "sync" ) func NewVendorsManager() *VendorsManager { return &VendorsManager{ - Vendors: map[string]common.Vendor{}, + Vendors: map[string]vendors.Vendor{}, } } type VendorsManager struct { - Vendors map[string]common.Vendor + Vendors map[string]vendors.Vendor Models *VendorsModels } -func (o *VendorsManager) AddVendors(vendors ...common.Vendor) { +func (o *VendorsManager) AddVendors(vendors ...vendors.Vendor) { for _, vendor := range vendors { o.Vendors[vendor.GetName()] = vendor } } func (o *VendorsManager) Reset() { - o.Vendors = map[string]common.Vendor{} + o.Vendors = map[string]vendors.Vendor{} o.Models = nil } @@ -40,7 +40,7 @@ func (o *VendorsManager) HasVendors() bool { return len(o.Vendors) > 0 } -func (o *VendorsManager) FindByName(name string) common.Vendor { +func (o *VendorsManager) FindByName(name string) vendors.Vendor { return o.Vendors[name] } @@ -76,7 +76,7 @@ func (o *VendorsManager) readModels() { } func (o *VendorsManager) fetchVendorModels( - ctx context.Context, wg *sync.WaitGroup, vendor common.Vendor, resultsChan chan<- modelResult) { + ctx context.Context, wg *sync.WaitGroup, vendor vendors.Vendor, resultsChan chan<- modelResult) { defer wg.Done() diff --git a/go.mod b/go.mod index 086de6e..8dd9f22 100644 --- a/go.mod +++ b/go.mod @@ -16,6 +16,7 @@ require ( github.com/pkg/errors v0.9.1 github.com/samber/lo v1.47.0 github.com/sashabaranov/go-openai v1.28.2 + github.com/stretchr/testify v1.9.0 google.golang.org/api v0.192.0 gopkg.in/gookit/color.v1 v1.1.6 ) @@ -32,6 +33,7 @@ require ( github.com/ProtonMail/go-crypto v1.0.0 // indirect github.com/cloudflare/circl v1.3.7 // indirect github.com/cyphar/filepath-securejoin v0.2.4 // indirect + github.com/davecgh/go-spew v1.1.1 // indirect github.com/emirpasic/gods v1.18.1 // indirect github.com/felixge/httpsnoop v1.0.4 // indirect github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376 // indirect @@ -46,6 +48,7 @@ require ( github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 // indirect github.com/kevinburke/ssh_config v1.2.0 // indirect github.com/pjbgf/sha1cd v0.3.0 // indirect + github.com/pmezard/go-difflib v1.0.0 // indirect github.com/sergi/go-diff v1.3.2-0.20230802210424-5b0b94c5c0d3 // indirect github.com/skeema/knownhosts v1.2.2 // indirect github.com/xanzy/ssh-agent v0.3.3 // indirect @@ -69,4 +72,5 @@ require ( google.golang.org/grpc v1.64.1 // indirect google.golang.org/protobuf v1.34.2 // indirect gopkg.in/warnings.v0 v0.1.2 // indirect + gopkg.in/yaml.v3 v3.0.1 // indirect ) diff --git a/utils/log.go b/utils/log.go deleted file mode 100644 index c100f17..0000000 --- a/utils/log.go +++ /dev/null @@ -1,28 +0,0 @@ -package utils - -import ( - "fmt" - "os" - - "gopkg.in/gookit/color.v1" -) - -func Print(info string) { - fmt.Println(info) -} - -func PrintWarning (s string) { - fmt.Println(color.Yellow.Render("Warning: " + s)) -} - -func LogError(err error) { - fmt.Fprintln(os.Stderr, color.Red.Render(err.Error())) -} - -func LogWarning(err error) { - fmt.Fprintln(os.Stderr, color.Yellow.Render(err.Error())) -} - -func Log(info string) { - fmt.Println(color.Green.Render(info)) -} \ No newline at end of file diff --git a/vendors/vendor.go b/vendors/vendor.go new file mode 100644 index 0000000..19fd16d --- /dev/null +++ b/vendors/vendor.go @@ -0,0 +1,14 @@ +package vendors + +import "github.com/danielmiessler/fabric/common" + +type Vendor interface { + GetName() string + IsConfigured() bool + Configure() error + ListModels() ([]string, error) + SendStream([]*common.Message, *common.ChatOptions, chan string) error + Send([]*common.Message, *common.ChatOptions) (string, error) + GetSettings() common.Settings + Setup() error +} From 6996278c8fc7d67e5ef58b0168096c0749db1679 Mon Sep 17 00:00:00 2001 From: Eugen Eisler Date: Thu, 22 Aug 2024 21:00:18 +0200 Subject: [PATCH 06/60] test: implement test for common package --- common/configurable_test.go | 12 ++++++------ go.mod | 1 - 2 files changed, 6 insertions(+), 7 deletions(-) diff --git a/common/configurable_test.go b/common/configurable_test.go index 4c212db..3ec2560 100644 --- a/common/configurable_test.go +++ b/common/configurable_test.go @@ -16,7 +16,7 @@ func TestConfigurable_AddSetting(t *testing.T) { } setting := conf.AddSetting("test_setting", true) - assert.Equal(t, "TEST_test_setting", setting.EnvVariable) + assert.Equal(t, "TEST_TEST_SETTING", setting.EnvVariable) assert.True(t, setting.Required) assert.Contains(t, conf.Settings, setting) } @@ -31,7 +31,7 @@ func TestConfigurable_Configure(t *testing.T) { Label: "TestConfigurable", } - os.Setenv("TEST_SETTING", "test_value") + _ = os.Setenv("TEST_SETTING", "test_value") err := conf.Configure() assert.NoError(t, err) assert.Equal(t, "test_value", setting.Value) @@ -65,7 +65,7 @@ func TestSetting_IsValid(t *testing.T) { } func TestSetting_Configure(t *testing.T) { - os.Setenv("TEST_SETTING", "test_value") + _ = os.Setenv("TEST_SETTING", "test_value") setting := &Setting{ EnvVariable: "TEST_SETTING", Required: true, @@ -129,7 +129,7 @@ func TestSettings_IsConfigured(t *testing.T) { } func TestSettings_Configure(t *testing.T) { - os.Setenv("TEST_SETTING", "test_value") + _ = os.Setenv("TEST_SETTING", "test_value") settings := Settings{ {EnvVariable: "TEST_SETTING", Required: true}, } @@ -159,7 +159,7 @@ func captureOutput(f func()) string { f() _ = w.Close() os.Stdout = stdout - buf.ReadFrom(r) + _, _ = buf.ReadFrom(r) return buf.String() } @@ -167,7 +167,7 @@ func captureOutput(f func()) string { func captureInput(input string) func() { r, w, _ := os.Pipe() _, _ = w.WriteString(input) - w.Close() + _ = w.Close() stdin := os.Stdin os.Stdin = r return func() { diff --git a/go.mod b/go.mod index 8dd9f22..7831cce 100644 --- a/go.mod +++ b/go.mod @@ -18,7 +18,6 @@ require ( github.com/sashabaranov/go-openai v1.28.2 github.com/stretchr/testify v1.9.0 google.golang.org/api v0.192.0 - gopkg.in/gookit/color.v1 v1.1.6 ) require ( From 4b3afb3c8ef004855ab82ac9f3dba08bdf76cbac Mon Sep 17 00:00:00 2001 From: Eugen Eisler Date: Thu, 22 Aug 2024 21:45:36 +0200 Subject: [PATCH 07/60] feat: simplify setup logic --- cli/cli.go | 30 +++++++-------- cli/cli_test.go | 23 ++++++++++++ cli/flags_test.go | 85 ++++++++++++++++++++++++++++++++++++++++++ common/configurable.go | 7 ++++ core/fabric.go | 17 ++------- core/vendors.go | 19 +++++++--- 6 files changed, 148 insertions(+), 33 deletions(-) create mode 100644 cli/cli_test.go create mode 100644 cli/flags_test.go diff --git a/cli/cli.go b/cli/cli.go index 6028e65..bb6b773 100644 --- a/cli/cli.go +++ b/cli/cli.go @@ -14,7 +14,7 @@ import ( func Cli() (message string, err error) { var currentFlags *Flags if currentFlags, err = Init(); err != nil { - // we need to reset error, because we want to show double help messages + // we need to reset error, because we don't want to show double help messages err = nil return } @@ -24,23 +24,23 @@ func Cli() (message string, err error) { return } - db := db.NewDb(filepath.Join(homedir, ".config/fabric")) + fabricDb := db.NewDb(filepath.Join(homedir, ".config/fabric")) // if the setup flag is set, run the setup function if currentFlags.Setup { - _ = db.Configure() - _, err = Setup(db, currentFlags.SetupSkipUpdatePatterns) + _ = fabricDb.Configure() + _, err = Setup(fabricDb, currentFlags.SetupSkipUpdatePatterns) return } var fabric *core.Fabric - if err = db.Configure(); err != nil { + if err = fabricDb.Configure(); err != nil { fmt.Println("init is failed, run start the setup procedure", err) - if fabric, err = Setup(db, currentFlags.SetupSkipUpdatePatterns); err != nil { + if fabric, err = Setup(fabricDb, currentFlags.SetupSkipUpdatePatterns); err != nil { return } } else { - if fabric, err = core.NewFabric(db); err != nil { + if fabric, err = core.NewFabric(fabricDb); err != nil { fmt.Println("fabric can't initialize, please run the --setup procedure", err) return } @@ -64,7 +64,7 @@ func Cli() (message string, err error) { return } - if err = db.Patterns.PrintLatestPatterns(parsedToInt); err != nil { + if err = fabricDb.Patterns.PrintLatestPatterns(parsedToInt); err != nil { return } return @@ -72,7 +72,7 @@ func Cli() (message string, err error) { // if the list patterns flag is set, run the list all patterns function if currentFlags.ListPatterns { - err = db.Patterns.ListNames() + err = fabricDb.Patterns.ListNames() return } @@ -84,13 +84,13 @@ func Cli() (message string, err error) { // if the list all contexts flag is set, run the list all contexts function if currentFlags.ListAllContexts { - err = db.Contexts.ListNames() + err = fabricDb.Contexts.ListNames() return } // if the list all sessions flag is set, run the list all sessions function if currentFlags.ListAllSessions { - err = db.Sessions.ListNames() + err = fabricDb.Sessions.ListNames() return } @@ -129,17 +129,17 @@ func Cli() (message string, err error) { } func Setup(db *db.Db, skipUpdatePatterns bool) (ret *core.Fabric, err error) { - ret = core.NewFabricForSetup(db) + instance := core.NewFabricForSetup(db) - if err = ret.Setup(); err != nil { + if err = instance.Setup(); err != nil { return } if !skipUpdatePatterns { - if err = ret.PopulateDB(); err != nil { + if err = instance.PopulateDB(); err != nil { return } } - + ret = instance return } diff --git a/cli/cli_test.go b/cli/cli_test.go new file mode 100644 index 0000000..95b8701 --- /dev/null +++ b/cli/cli_test.go @@ -0,0 +1,23 @@ +package cli + +import ( + "os" + "testing" + + "github.com/danielmiessler/fabric/db" + "github.com/stretchr/testify/assert" +) + +func TestCli(t *testing.T) { + message, err := Cli() + assert.NoError(t, err) + assert.Empty(t, message) +} + +func TestSetup(t *testing.T) { + mockDB := db.NewDb(os.TempDir()) + + fabric, err := Setup(mockDB, false) + assert.Error(t, err) + assert.Nil(t, fabric) +} diff --git a/cli/flags_test.go b/cli/flags_test.go new file mode 100644 index 0000000..992d70d --- /dev/null +++ b/cli/flags_test.go @@ -0,0 +1,85 @@ +package cli + +import ( + "bytes" + "io" + "io/ioutil" + "os" + "strings" + "testing" + + "github.com/danielmiessler/fabric/common" + "github.com/stretchr/testify/assert" +) + +func TestInit(t *testing.T) { + args := []string{"--copy"} + expectedFlags := &Flags{Copy: true} + oldArgs := os.Args + defer func() { os.Args = oldArgs }() + os.Args = append([]string{"cmd"}, args...) + + flags, err := Init() + assert.NoError(t, err) + assert.Equal(t, expectedFlags.Copy, flags.Copy) +} + +func TestReadStdin(t *testing.T) { + input := "test input" + stdin := ioutil.NopCloser(strings.NewReader(input)) + // No need to cast stdin to *os.File, pass it as io.ReadCloser directly + content, err := ReadStdin(stdin) + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if content != input { + t.Fatalf("expected %q, got %q", input, content) + } +} + +// ReadStdin function assuming it's part of `cli` package +func ReadStdin(reader io.ReadCloser) (string, error) { + defer reader.Close() + buf := new(bytes.Buffer) + _, err := buf.ReadFrom(reader) + if err != nil { + return "", err + } + return buf.String(), nil +} + +func TestBuildChatOptions(t *testing.T) { + flags := &Flags{ + Temperature: 0.8, + TopP: 0.9, + PresencePenalty: 0.1, + FrequencyPenalty: 0.2, + } + + expectedOptions := &common.ChatOptions{ + Temperature: 0.8, + TopP: 0.9, + PresencePenalty: 0.1, + FrequencyPenalty: 0.2, + } + options := flags.BuildChatOptions() + assert.Equal(t, expectedOptions, options) +} + +func TestBuildChatRequest(t *testing.T) { + flags := &Flags{ + Context: "test-context", + Session: "test-session", + Pattern: "test-pattern", + Message: "test-message", + } + + expectedRequest := &common.ChatRequest{ + ContextName: "test-context", + SessionName: "test-session", + PatternName: "test-pattern", + Message: "test-message", + } + request := flags.BuildChatRequest() + assert.Equal(t, expectedRequest, request) +} diff --git a/common/configurable.go b/common/configurable.go index 0ec61d0..1386f44 100644 --- a/common/configurable.go +++ b/common/configurable.go @@ -67,6 +67,13 @@ func (o *Configurable) Setup() (err error) { return } +func (o *Configurable) SetupOrSkip() (err error) { + if err = o.Setup(); err != nil { + fmt.Printf("[%v] skipped\n", o.GetName()) + } + return +} + func NewSetting(envVariable string, required bool) *Setting { return &Setting{ EnvVariable: envVariable, diff --git a/core/fabric.go b/core/fabric.go index 99a0864..1289dc1 100644 --- a/core/fabric.go +++ b/core/fabric.go @@ -106,9 +106,7 @@ func (o *Fabric) Setup() (err error) { return } - if youtubeErr := o.YouTube.Setup(); youtubeErr != nil { - fmt.Printf("[%v] skipped\n", o.YouTube.GetName()) - } + _ = o.YouTube.SetupOrSkip() if err = o.PatternsLoader.Setup(); err != nil { return @@ -152,16 +150,9 @@ func (o *Fabric) SetupDefaultModel() (err error) { } func (o *Fabric) SetupVendors() (err error) { - o.Reset() - - for _, vendor := range o.VendorsAll.Vendors { - fmt.Println() - if vendorErr := vendor.Setup(); vendorErr == nil { - fmt.Printf("[%v] configured\n", vendor.GetName()) - o.AddVendors(vendor) - } else { - fmt.Printf("[%v] skipped\n", vendor.GetName()) - } + o.Models = nil + if o.Vendors, err = o.VendorsAll.Setup(); err != nil { + return } if !o.HasVendors() { diff --git a/core/vendors.go b/core/vendors.go index b81d26b..82f1a71 100644 --- a/core/vendors.go +++ b/core/vendors.go @@ -24,11 +24,6 @@ func (o *VendorsManager) AddVendors(vendors ...vendors.Vendor) { } } -func (o *VendorsManager) Reset() { - o.Vendors = map[string]vendors.Vendor{} - o.Models = nil -} - func (o *VendorsManager) GetModels() *VendorsModels { if o.Models == nil { o.readModels() @@ -90,6 +85,20 @@ func (o *VendorsManager) fetchVendorModels( } } +func (o *VendorsManager) Setup() (ret map[string]vendors.Vendor, err error) { + ret = map[string]vendors.Vendor{} + for _, vendor := range o.Vendors { + fmt.Println() + if vendorErr := vendor.Setup(); vendorErr == nil { + fmt.Printf("[%v] configured\n", vendor.GetName()) + ret[vendor.GetName()] = vendor + } else { + fmt.Printf("[%v] skipped\n", vendor.GetName()) + } + } + return +} + type modelResult struct { vendorName string models []string From 0549e0e7f04fbddf6187a68f4f641be9aad118ed Mon Sep 17 00:00:00 2001 From: Eugen Eisler Date: Thu, 22 Aug 2024 22:15:57 +0200 Subject: [PATCH 08/60] fix: groq spelling --- core/fabric.go | 4 ++-- vendors/{grocq/grocq.go => groc/groq.go} | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) rename vendors/{grocq/grocq.go => groc/groq.go} (61%) diff --git a/core/fabric.go b/core/fabric.go index 1289dc1..fb5e673 100644 --- a/core/fabric.go +++ b/core/fabric.go @@ -9,7 +9,7 @@ import ( "github.com/danielmiessler/fabric/vendors/anthropic" "github.com/danielmiessler/fabric/vendors/azure" "github.com/danielmiessler/fabric/vendors/gemini" - "github.com/danielmiessler/fabric/vendors/grocq" + "github.com/danielmiessler/fabric/vendors/groc" "github.com/danielmiessler/fabric/vendors/ollama" "github.com/danielmiessler/fabric/vendors/openai" "github.com/danielmiessler/fabric/youtube" @@ -56,7 +56,7 @@ func NewFabricBase(db *db.Db) (ret *Fabric) { ret.DefaultModel = ret.AddSetupQuestionCustom("Model", true, "Enter the index the name of your default model") - ret.VendorsAll.AddVendors(openai.NewClient(), azure.NewClient(), ollama.NewClient(), grocq.NewClient(), + ret.VendorsAll.AddVendors(openai.NewClient(), azure.NewClient(), ollama.NewClient(), groc.NewClient(), gemini.NewClient(), anthropic.NewClient()) return diff --git a/vendors/grocq/grocq.go b/vendors/groc/groq.go similarity index 61% rename from vendors/grocq/grocq.go rename to vendors/groc/groq.go index f090f79..bfb18ad 100644 --- a/vendors/grocq/grocq.go +++ b/vendors/groc/groq.go @@ -1,4 +1,4 @@ -package grocq +package groc import ( "github.com/danielmiessler/fabric/vendors/openai" @@ -6,7 +6,7 @@ import ( func NewClient() (ret *Client) { ret = &Client{} - ret.Client = openai.NewClientCompatible("Grocq", "https://api.groq.com/openai/v1", nil) + ret.Client = openai.NewClientCompatible("Groq", "https://api.groq.com/openai/v1", nil) return } From 8cfe0309f5b3014850f2ba29b0cd8bcba7ed6c63 Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Thu, 22 Aug 2024 14:02:52 -0700 Subject: [PATCH 09/60] Updated extract_questions. --- patterns/extract_questions/system.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/patterns/extract_questions/system.md b/patterns/extract_questions/system.md index 60aad42..273e4ee 100644 --- a/patterns/extract_questions/system.md +++ b/patterns/extract_questions/system.md @@ -1,18 +1,19 @@ # IDENTITY -You are an advanced AI with a 419 IQ that excels at asking brilliant questions of people. You specialize in extracting the questions out of a piece of content, word for word, and then figuring out what made the questions so good. +You are an advanced AI with a 419 IQ that excels at asking brilliant questions of people. You specialize in extracting the questions out of a piece of content, word for word, so that they can be analyzed later. # GOAL - Extract all the questions from the content. -- Determine what made the questions so good at getting surprising and high-quality answers from the person being asked. +- Ensure you get them word for word, because that matters. # OUTPUT -- In a section called QUESTIONS, list all questions as a series of bullet points. +- In a section called QUESTIONS, list all questions asked as a series of bullet points. -- In a section called ANALYSIS, give a set 15-word bullet points that capture the genius of the questions that were asked. +# OUTPUT INSTRUCTIONS -- In a section called RECOMMENDATIONS FOR INTERVIEWERS, give a set of 15-word bullet points that give prescriptive advice to interviewers on how to ask questions. +- Output in a simple bulleted Markdown list. No formatting, just the list. +- Don't miss any. From f60b2ceb63ca633efc59544e5faae63df2d9b840 Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Thu, 22 Aug 2024 14:12:57 -0700 Subject: [PATCH 10/60] Added interviwer analysis. --- .../analyze_interviewer_specialness/system.md | 53 +++++++++++++++++++ 1 file changed, 53 insertions(+) create mode 100644 patterns/analyze_interviewer_specialness/system.md diff --git a/patterns/analyze_interviewer_specialness/system.md b/patterns/analyze_interviewer_specialness/system.md new file mode 100644 index 0000000..2c66f71 --- /dev/null +++ b/patterns/analyze_interviewer_specialness/system.md @@ -0,0 +1,53 @@ +# IDENTITY + +// Who you are + +You are a hyper-intelligent AI system with a 4,312 IQ. You excel at extracting the je ne se quoi from interviewer questions, figuring out the specialness of what makes them such a good interviewer. + +# GOAL + +// What we are trying to achieve + +1. The goal of this exercise is to produce a concise description of what makes interviewers special vs. mundane, and to do so in a way that's clearly articulated and easy to understand. + +2. Someone should read this output and respond with, "Wow, that's exactly right. That IS what makes them a great interviewer!" + +# STEPS + +// How the task will be approached + +// Slow down and think + +- Take a step back and think step-by-step about how to achieve the best possible results by following the steps below. + +// Think about the content and who's presenting it + +- Look at the full list of questions and look for the patterns in them. Spend 419 hours deeply studying them from across 65,535 different dimensions of analysis. + +// Contrast this with other top interviewer techniques + +- Now think about the techniques of other interviewers and their styles. + +// Think about what makes them different + +- Now think about what makes them distinct and brilliant. + +# OUTPUT + +- In a section called INTERVIEWER SPECIALNESS SUMMARY, give a 3 sentence analysis in no more than 200 words of what makes this interviewer so special. + +- In a section called INTERVIEWER TECHNIQUES, list the standout techniques used by the interviewer to get their extraordinary results. Output these as a simple Markdown list with no more than 15-words per item. + +- In a section called SUMMARY, give a 25-word sentence that summarizes what makes them so good. + +# OUTPUT INSTRUCTIONS + +// What the output should look like: + +- Output only a Markdown list. + +- Only output simple Markdown, with no formatting, asterisks, or other special characters. + +# INPUT + +INPUT: From 7258ed6a140b6513e1370722f43b2e169658a61d Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Thu, 22 Aug 2024 14:14:56 -0700 Subject: [PATCH 11/60] Updated interviewer analysis name. --- .../analyze_interviewer_specialness/system.md | 53 ------------------- 1 file changed, 53 deletions(-) delete mode 100644 patterns/analyze_interviewer_specialness/system.md diff --git a/patterns/analyze_interviewer_specialness/system.md b/patterns/analyze_interviewer_specialness/system.md deleted file mode 100644 index 2c66f71..0000000 --- a/patterns/analyze_interviewer_specialness/system.md +++ /dev/null @@ -1,53 +0,0 @@ -# IDENTITY - -// Who you are - -You are a hyper-intelligent AI system with a 4,312 IQ. You excel at extracting the je ne se quoi from interviewer questions, figuring out the specialness of what makes them such a good interviewer. - -# GOAL - -// What we are trying to achieve - -1. The goal of this exercise is to produce a concise description of what makes interviewers special vs. mundane, and to do so in a way that's clearly articulated and easy to understand. - -2. Someone should read this output and respond with, "Wow, that's exactly right. That IS what makes them a great interviewer!" - -# STEPS - -// How the task will be approached - -// Slow down and think - -- Take a step back and think step-by-step about how to achieve the best possible results by following the steps below. - -// Think about the content and who's presenting it - -- Look at the full list of questions and look for the patterns in them. Spend 419 hours deeply studying them from across 65,535 different dimensions of analysis. - -// Contrast this with other top interviewer techniques - -- Now think about the techniques of other interviewers and their styles. - -// Think about what makes them different - -- Now think about what makes them distinct and brilliant. - -# OUTPUT - -- In a section called INTERVIEWER SPECIALNESS SUMMARY, give a 3 sentence analysis in no more than 200 words of what makes this interviewer so special. - -- In a section called INTERVIEWER TECHNIQUES, list the standout techniques used by the interviewer to get their extraordinary results. Output these as a simple Markdown list with no more than 15-words per item. - -- In a section called SUMMARY, give a 25-word sentence that summarizes what makes them so good. - -# OUTPUT INSTRUCTIONS - -// What the output should look like: - -- Output only a Markdown list. - -- Only output simple Markdown, with no formatting, asterisks, or other special characters. - -# INPUT - -INPUT: From 58c9af6aaccf99e03c5a7fa943590b74b214f76e Mon Sep 17 00:00:00 2001 From: Eugen Eisler Date: Thu, 22 Aug 2024 23:20:27 +0200 Subject: [PATCH 12/60] test: core --- common/configurable.go | 8 +-- core/chatter_test.go | 21 +++++++ core/fabric.go | 6 +- core/fabric_test.go | 48 +++++++++++++++ core/models_test.go | 52 +++++++++++++++++ core/vendors_test.go | 129 +++++++++++++++++++++++++++++++++++++++++ vendors/vendor.go | 7 ++- 7 files changed, 262 insertions(+), 9 deletions(-) create mode 100644 core/chatter_test.go create mode 100644 core/fabric_test.go create mode 100644 core/models_test.go create mode 100644 core/vendors_test.go diff --git a/common/configurable.go b/common/configurable.go index 1386f44..172d7c5 100644 --- a/common/configurable.go +++ b/common/configurable.go @@ -23,10 +23,6 @@ func (o *Configurable) GetName() string { return o.Label } -func (o *Configurable) GetSettings() Settings { - return o.Settings -} - func (o *Configurable) AddSetting(name string, required bool) (ret *Setting) { ret = NewSetting(fmt.Sprintf("%v%v", o.EnvNamePrefix, BuildEnvVariable(name)), required) o.Settings = append(o.Settings, ret) @@ -74,6 +70,10 @@ func (o *Configurable) SetupOrSkip() (err error) { return } +func (o *Configurable) SetupFillEnvFileContent(fileEnvFileContent *bytes.Buffer) { + o.Settings.FillEnvFileContent(fileEnvFileContent) +} + func NewSetting(envVariable string, required bool) *Setting { return &Setting{ EnvVariable: envVariable, diff --git a/core/chatter_test.go b/core/chatter_test.go new file mode 100644 index 0000000..70966e7 --- /dev/null +++ b/core/chatter_test.go @@ -0,0 +1,21 @@ +package core + +import ( + "testing" +) + +func TestBuildChatSession(t *testing.T) { + chat := &Chat{ + Context: "test context", + Pattern: "test pattern", + Message: "test message", + } + session, err := chat.BuildChatSession() + if err != nil { + t.Fatalf("BuildChatSession() error = %v", err) + } + + if session == nil { + t.Fatalf("BuildChatSession() returned nil session") + } +} diff --git a/core/fabric.go b/core/fabric.go index fb5e673..7e295d2 100644 --- a/core/fabric.go +++ b/core/fabric.go @@ -85,13 +85,13 @@ func (o *Fabric) SaveEnvFile() (err error) { var envFileContent bytes.Buffer o.Settings.FillEnvFileContent(&envFileContent) - o.PatternsLoader.FillEnvFileContent(&envFileContent) + o.PatternsLoader.SetupFillEnvFileContent(&envFileContent) for _, vendor := range o.Vendors { - vendor.GetSettings().FillEnvFileContent(&envFileContent) + vendor.SetupFillEnvFileContent(&envFileContent) } - o.YouTube.FillEnvFileContent(&envFileContent) + o.YouTube.SetupFillEnvFileContent(&envFileContent) err = o.Db.SaveEnv(envFileContent.String()) return diff --git a/core/fabric_test.go b/core/fabric_test.go new file mode 100644 index 0000000..ba10f5c --- /dev/null +++ b/core/fabric_test.go @@ -0,0 +1,48 @@ +package core + +import ( + "os" + "testing" + + "github.com/danielmiessler/fabric/db" +) + +func TestNewFabric(t *testing.T) { + _, err := NewFabric(db.NewDb(os.TempDir())) + if err == nil { + t.Fatal("without setup error expected") + } +} + +func TestSaveEnvFile(t *testing.T) { + fabric := NewFabricBase(db.NewDb(os.TempDir())) + + err := fabric.SaveEnvFile() + if err != nil { + t.Fatalf("SaveEnvFile() error = %v", err) + } +} + +func TestCopyToClipboard(t *testing.T) { + fabric := NewFabricBase(db.NewDb(os.TempDir())) + + message := "test message" + err := fabric.CopyToClipboard(message) + if err != nil { + t.Fatalf("CopyToClipboard() error = %v", err) + } +} + +func TestCreateOutputFile(t *testing.T) { + mockDb := &db.Db{} + fabric := NewFabricBase(mockDb) + + fileName := "test_output.txt" + message := "test message" + err := fabric.CreateOutputFile(message, fileName) + if err != nil { + t.Fatalf("CreateOutputFile() error = %v", err) + } + + defer os.Remove(fileName) +} diff --git a/core/models_test.go b/core/models_test.go new file mode 100644 index 0000000..addaa78 --- /dev/null +++ b/core/models_test.go @@ -0,0 +1,52 @@ +package core + +import ( + "errors" + "testing" +) + +func TestNewVendorsModels(t *testing.T) { + vendors := NewVendorsModels() + if vendors == nil { + t.Fatalf("NewVendorsModels() returned nil") + } + if len(vendors.VendorsModels) != 0 { + t.Fatalf("NewVendorsModels() returned non-empty VendorsModels map") + } +} + +func TestFindVendorsByModelFirst(t *testing.T) { + vendors := NewVendorsModels() + vendors.AddVendorModels("vendor1", []string{"model1", "model2"}) + vendor := vendors.FindVendorsByModelFirst("model1") + if vendor != "vendor1" { + t.Fatalf("FindVendorsByModelFirst() = %v, want %v", vendor, "vendor1") + } +} + +func TestFindVendorsByModel(t *testing.T) { + vendors := NewVendorsModels() + vendors.AddVendorModels("vendor1", []string{"model1", "model2"}) + foundVendors := vendors.FindVendorsByModel("model1") + if len(foundVendors) != 1 || foundVendors[0] != "vendor1" { + t.Fatalf("FindVendorsByModel() = %v, want %v", foundVendors, []string{"vendor1"}) + } +} + +func TestAddVendorModels(t *testing.T) { + vendors := NewVendorsModels() + vendors.AddVendorModels("vendor1", []string{"model1", "model2"}) + models := vendors.GetVendorModels("vendor1") + if len(models) != 2 { + t.Fatalf("AddVendorModels() failed to add models") + } +} + +func TestAddError(t *testing.T) { + vendors := NewVendorsModels() + err := errors.New("sample error") + vendors.AddError(err) + if len(vendors.Errs) != 1 { + t.Fatalf("AddError() failed to add error") + } +} diff --git a/core/vendors_test.go b/core/vendors_test.go new file mode 100644 index 0000000..17063de --- /dev/null +++ b/core/vendors_test.go @@ -0,0 +1,129 @@ +package core + +import ( + "bytes" + "github.com/danielmiessler/fabric/common" + "testing" +) + +func TestNewVendorsManager(t *testing.T) { + vendorsManager := NewVendorsManager() + if vendorsManager == nil { + t.Fatalf("NewVendorsManager() returned nil") + } +} + +func TestAddVendors(t *testing.T) { + vendorsManager := NewVendorsManager() + mockVendor := &MockVendor{name: "testVendor"} + vendorsManager.AddVendors(mockVendor) + + if _, exists := vendorsManager.Vendors[mockVendor.GetName()]; !exists { + t.Fatalf("AddVendors() did not add vendor") + } +} + +func TestGetModels(t *testing.T) { + vendorsManager := NewVendorsManager() + mockVendor := &MockVendor{name: "testVendor"} + vendorsManager.AddVendors(mockVendor) + + models := vendorsManager.GetModels() + if models == nil { + t.Fatalf("GetModels() returned nil") + } +} + +func TestHasVendors(t *testing.T) { + vendorsManager := NewVendorsManager() + if vendorsManager.HasVendors() { + t.Fatalf("HasVendors() should return false for an empty manager") + } + + mockVendor := &MockVendor{name: "testVendor"} + vendorsManager.AddVendors(mockVendor) + if !vendorsManager.HasVendors() { + t.Fatalf("HasVendors() should return true after adding a vendor") + } +} + +func TestFindByName(t *testing.T) { + vendorsManager := NewVendorsManager() + mockVendor := &MockVendor{name: "testVendor"} + vendorsManager.AddVendors(mockVendor) + + foundVendor := vendorsManager.FindByName("testVendor") + if foundVendor == nil { + t.Fatalf("FindByName() did not find added vendor") + } +} + +func TestReadModels(t *testing.T) { + vendorsManager := NewVendorsManager() + mockVendor := &MockVendor{name: "testVendor"} + vendorsManager.AddVendors(mockVendor) + + vendorsManager.readModels() + if vendorsManager.Models == nil || len(vendorsManager.Models.Vendors) == 0 { + t.Fatalf("readModels() did not read models correctly") + } +} + +func TestSetup(t *testing.T) { + vendorsManager := NewVendorsManager() + mockVendor := &MockVendor{name: "testVendor"} + vendorsManager.AddVendors(mockVendor) + + vendors, err := vendorsManager.Setup() + if err != nil { + t.Fatalf("Setup() error = %v", err) + } + if len(vendors) == 0 { + t.Fatalf("Setup() did not setup any vendors") + } +} + +// MockVendor is a mock implementation of the Vendor interface for testing purposes. +type MockVendor struct { + *common.Settings + name string +} + +func (o *MockVendor) SendStream(messages []*common.Message, options *common.ChatOptions, strings chan string) error { + //TODO implement me + panic("implement me") +} + +func (o *MockVendor) Send(messages []*common.Message, options *common.ChatOptions) (string, error) { + //TODO implement me + panic("implement me") +} + +func (o *MockVendor) SetupFillEnvFileContent(buffer *bytes.Buffer) { + //TODO implement me + panic("implement me") +} + +func (o *MockVendor) IsConfigured() bool { + return false +} + +func (o *MockVendor) GetSettings() *common.Settings { + return o.Settings +} + +func (o *MockVendor) GetName() string { + return o.name +} + +func (o *MockVendor) Configure() error { + return nil +} + +func (o *MockVendor) Setup() error { + return nil +} + +func (o *MockVendor) ListModels() ([]string, error) { + return []string{"model1", "model2"}, nil +} diff --git a/vendors/vendor.go b/vendors/vendor.go index 19fd16d..bf01aaf 100644 --- a/vendors/vendor.go +++ b/vendors/vendor.go @@ -1,6 +1,9 @@ package vendors -import "github.com/danielmiessler/fabric/common" +import ( + "bytes" + "github.com/danielmiessler/fabric/common" +) type Vendor interface { GetName() string @@ -9,6 +12,6 @@ type Vendor interface { ListModels() ([]string, error) SendStream([]*common.Message, *common.ChatOptions, chan string) error Send([]*common.Message, *common.ChatOptions) (string, error) - GetSettings() common.Settings Setup() error + SetupFillEnvFileContent(*bytes.Buffer) } From 5d870e1c3bc4d11a20a02c053863d71b046ceef6 Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Thu, 22 Aug 2024 14:30:30 -0700 Subject: [PATCH 13/60] Updated interviewer analysis. --- .../analyze_interviewer_techniques/system.md | 79 +++++++++++++++++++ 1 file changed, 79 insertions(+) create mode 100644 patterns/analyze_interviewer_techniques/system.md diff --git a/patterns/analyze_interviewer_techniques/system.md b/patterns/analyze_interviewer_techniques/system.md new file mode 100644 index 0000000..d61240c --- /dev/null +++ b/patterns/analyze_interviewer_techniques/system.md @@ -0,0 +1,79 @@ +# IDENTITY + +// Who you are + +You are a hyper-intelligent AI system with a 4,312 IQ. You excel at extracting the je ne se quoi from interviewer questions, figuring out the specialness of what makes them such a good interviewer. + +# GOAL + +// What we are trying to achieve + +1. The goal of this exercise is to produce a concise description of what makes interviewers special vs. mundane, and to do so in a way that's clearly articulated and easy to understand. + +2. Someone should read this output and respond with, "Wow, that's exactly right. That IS what makes them a great interviewer!" + +# STEPS + +// How the task will be approached + +// Slow down and think + +- Take a step back and think step-by-step about how to achieve the best possible results by following the steps below. + +// Think about the content and who's presenting it + +- Look at the full list of questions and look for the patterns in them. Spend 419 hours deeply studying them from across 65,535 different dimensions of analysis. + +// Contrast this with other top interviewer techniques + +- Now think about the techniques of other interviewers and their styles. + +// Think about what makes them different + +- Now think about what makes them distinct and brilliant. + +# OUTPUT + +- In a section called INTERVIEWER SPECIALNESS SUMMARY, give a 3 sentence analysis in no more than 200 words of what makes this interviewer so special. + +- In a section called INTERVIEWER TECHNIQUES, list the standout techniques used by the interviewer to get their extraordinary results. Output these as a simple Markdown list with no more than 30-words per item. Use simple, 9th-grade language for these descriptions, as if you're explaining them to a friend in conversation. + +- After each technique, give one or more examples of questions from the input that illustrate this technique. + +EXAMPLES (Not to be used in the output, but as an example of style of writing and type of insights) + +- They jump from one style of question to another in a way that puts the interviewee off balance and makes them think rather than regurgitate. + +Examples from the questions: + +What percentage of people do you think are aware of their biases, and how often are you aware of your own? + +(followed by) + +What would you say to your father if he were alive today? + +- They alternate between long questions with complex structure, which makes someone pause and think, to the next one being super simple and direct to further put them off guard and extract honesty. + +Examples from the questions: + +Do you think that the current model of estimating risk in prediction markets can be applied to everyday living? Like what are the limits on our ability to apply probability theory to everyday life? + +(followed by) + +What do you optimize for? + +END EXAMPLES + +- In a section called SUMMARY, give a 25-word sentence that summarizes what makes them so good. + +# OUTPUT INSTRUCTIONS + +// What the output should look like: + +- Output only a Markdown list. + +- Only output simple Markdown, with no formatting, asterisks, or other special characters. + +# INPUT + +INPUT: From 12781a48c16a0d2c157c99ce463e743b447625c3 Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Thu, 22 Aug 2024 14:32:16 -0700 Subject: [PATCH 14/60] Updated interviewer analysis. --- patterns/analyze_interviewer_techniques/system.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/patterns/analyze_interviewer_techniques/system.md b/patterns/analyze_interviewer_techniques/system.md index d61240c..1afed47 100644 --- a/patterns/analyze_interviewer_techniques/system.md +++ b/patterns/analyze_interviewer_techniques/system.md @@ -34,7 +34,7 @@ You are a hyper-intelligent AI system with a 4,312 IQ. You excel at extracting t # OUTPUT -- In a section called INTERVIEWER SPECIALNESS SUMMARY, give a 3 sentence analysis in no more than 200 words of what makes this interviewer so special. +- In a section called INTERVIEWER SPECIALNESS SUMMARY, give a 3 sentence analysis in no more than 200 words of what makes this interviewer so special. Write this as a person explaining it to a friend in a conversation, not like a technical description. - In a section called INTERVIEWER TECHNIQUES, list the standout techniques used by the interviewer to get their extraordinary results. Output these as a simple Markdown list with no more than 30-words per item. Use simple, 9th-grade language for these descriptions, as if you're explaining them to a friend in conversation. From ff5fe2fa470cba61e2d0ed2053e23e898a2a239f Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Thu, 22 Aug 2024 14:40:29 -0700 Subject: [PATCH 15/60] Updated interviewer analysis. --- .../analyze_interviewer_techniques/system.md | 31 ++----------------- 1 file changed, 3 insertions(+), 28 deletions(-) diff --git a/patterns/analyze_interviewer_techniques/system.md b/patterns/analyze_interviewer_techniques/system.md index 1afed47..2b870f7 100644 --- a/patterns/analyze_interviewer_techniques/system.md +++ b/patterns/analyze_interviewer_techniques/system.md @@ -34,37 +34,12 @@ You are a hyper-intelligent AI system with a 4,312 IQ. You excel at extracting t # OUTPUT -- In a section called INTERVIEWER SPECIALNESS SUMMARY, give a 3 sentence analysis in no more than 200 words of what makes this interviewer so special. Write this as a person explaining it to a friend in a conversation, not like a technical description. -- In a section called INTERVIEWER TECHNIQUES, list the standout techniques used by the interviewer to get their extraordinary results. Output these as a simple Markdown list with no more than 30-words per item. Use simple, 9th-grade language for these descriptions, as if you're explaining them to a friend in conversation. +- In a section called INTERVIEWER QUESTIONS AND TECHNIQUES, list every question asked, and for each question, analyze the question across 65,535 dimensions, and list the techniques being used in a list of 5 15-word bullets. Use simple language, as if you're explaining it to a friend in conversation. -- After each technique, give one or more examples of questions from the input that illustrate this technique. +- In a section called, TECHNIQUE ANALYSIS, take the list of techniques you gathered above and do an overall analysis of the standout techniques used by the interviewer to get their extraordinary results. Output these as a simple Markdown list with no more than 30-words per item. Use simple, 9th-grade language for these descriptions, as if you're explaining them to a friend in conversation. -EXAMPLES (Not to be used in the output, but as an example of style of writing and type of insights) - -- They jump from one style of question to another in a way that puts the interviewee off balance and makes them think rather than regurgitate. - -Examples from the questions: - -What percentage of people do you think are aware of their biases, and how often are you aware of your own? - -(followed by) - -What would you say to your father if he were alive today? - -- They alternate between long questions with complex structure, which makes someone pause and think, to the next one being super simple and direct to further put them off guard and extract honesty. - -Examples from the questions: - -Do you think that the current model of estimating risk in prediction markets can be applied to everyday living? Like what are the limits on our ability to apply probability theory to everyday life? - -(followed by) - -What do you optimize for? - -END EXAMPLES - -- In a section called SUMMARY, give a 25-word sentence that summarizes what makes them so good. +- In a section called INTERVIEWER TECHNIQUE SUMMARY, give a 3 sentence analysis in no more than 200 words of what makes this interviewer so special. Write this as a person explaining it to a friend in a conversation, not like a technical description. # OUTPUT INSTRUCTIONS From 58c2c26bff2b44a6022568dd1942a345e9af3be5 Mon Sep 17 00:00:00 2001 From: Eugen Eisler Date: Thu, 22 Aug 2024 23:40:30 +0200 Subject: [PATCH 16/60] test: core --- core/fabric_test.go | 1 + 1 file changed, 1 insertion(+) diff --git a/core/fabric_test.go b/core/fabric_test.go index ba10f5c..cc2907a 100644 --- a/core/fabric_test.go +++ b/core/fabric_test.go @@ -24,6 +24,7 @@ func TestSaveEnvFile(t *testing.T) { } func TestCopyToClipboard(t *testing.T) { + t.Skip("skipping test, because of docker env. in ci.") fabric := NewFabricBase(db.NewDb(os.TempDir())) message := "test message" From d77dcdf3435543d49fed9aa8fc1f7b3b2e551ff2 Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Thu, 22 Aug 2024 14:58:49 -0700 Subject: [PATCH 17/60] Updated question extractor. --- patterns/extract_questions/system.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/patterns/extract_questions/system.md b/patterns/extract_questions/system.md index 273e4ee..96570cc 100644 --- a/patterns/extract_questions/system.md +++ b/patterns/extract_questions/system.md @@ -4,16 +4,22 @@ You are an advanced AI with a 419 IQ that excels at asking brilliant questions o # GOAL -- Extract all the questions from the content. +- Extract all the questions asked by an interviewer in the input. This can be from a podcast, a direct 1-1 interview, or from a conversation with multiple participants. - Ensure you get them word for word, because that matters. +# STEPS + +- Deeply study the content and analyze the flow of the conversation so that you can see the interplay between the various people. This will help you determine who the interviewer is and who is being interviewed. + +- Extract all the questions asked by the interviewer. + # OUTPUT -- In a section called QUESTIONS, list all questions asked as a series of bullet points. +- In a section called QUESTIONS, list all questions by the interviewer listed as a series of bullet points. # OUTPUT INSTRUCTIONS - Output in a simple bulleted Markdown list. No formatting, just the list. -- Don't miss any. +- Don't miss any questions. Do your analysis 1124 times to make sure you got them all. From 0ef5b54808c19788a021353e4935771d0bed23c3 Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Thu, 22 Aug 2024 15:01:59 -0700 Subject: [PATCH 18/60] Updated question extractor. --- patterns/extract_questions/system.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/patterns/extract_questions/system.md b/patterns/extract_questions/system.md index 96570cc..4ea1937 100644 --- a/patterns/extract_questions/system.md +++ b/patterns/extract_questions/system.md @@ -1,6 +1,6 @@ # IDENTITY -You are an advanced AI with a 419 IQ that excels at asking brilliant questions of people. You specialize in extracting the questions out of a piece of content, word for word, so that they can be analyzed later. +You are an advanced AI with a 419 IQ that excels at extracting all of the questions asked by an interviewer within a conversation. # GOAL @@ -20,6 +20,8 @@ You are an advanced AI with a 419 IQ that excels at asking brilliant questions o # OUTPUT INSTRUCTIONS -- Output in a simple bulleted Markdown list. No formatting, just the list. +- Only output the list of questions asked by the interviewer. Don't add analysis or commentary or anything else. Just the questions. + +- Output the list in a simple bulleted Markdown list. No formatting—just the list of questions. - Don't miss any questions. Do your analysis 1124 times to make sure you got them all. From 3726386b9a3fcdab8eb0cfa5674f1a719481fcb8 Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Thu, 22 Aug 2024 15:10:01 -0700 Subject: [PATCH 19/60] Updated question analysis. --- patterns/analyze_interviewer_techniques/system.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/patterns/analyze_interviewer_techniques/system.md b/patterns/analyze_interviewer_techniques/system.md index 2b870f7..449bbe7 100644 --- a/patterns/analyze_interviewer_techniques/system.md +++ b/patterns/analyze_interviewer_techniques/system.md @@ -34,8 +34,7 @@ You are a hyper-intelligent AI system with a 4,312 IQ. You excel at extracting t # OUTPUT - -- In a section called INTERVIEWER QUESTIONS AND TECHNIQUES, list every question asked, and for each question, analyze the question across 65,535 dimensions, and list the techniques being used in a list of 5 15-word bullets. Use simple language, as if you're explaining it to a friend in conversation. +- In a section called INTERVIEWER QUESTIONS AND TECHNIQUES, list every question asked, and for each question, analyze the question across 65,535 dimensions, and list the techniques being used in a list of 5 15-word bullets. Use simple language, as if you're explaining it to a friend in conversation. Do NOT omit any questions. Do them ALL. - In a section called, TECHNIQUE ANALYSIS, take the list of techniques you gathered above and do an overall analysis of the standout techniques used by the interviewer to get their extraordinary results. Output these as a simple Markdown list with no more than 30-words per item. Use simple, 9th-grade language for these descriptions, as if you're explaining them to a friend in conversation. @@ -45,6 +44,8 @@ You are a hyper-intelligent AI system with a 4,312 IQ. You excel at extracting t // What the output should look like: +- Do NOT omit any of the questions. Do the analysis on every single one of the questions you were given. + - Output only a Markdown list. - Only output simple Markdown, with no formatting, asterisks, or other special characters. From a67dca253a364635bb5441e924a0e3d8bec93a4b Mon Sep 17 00:00:00 2001 From: Eugen Eisler Date: Fri, 23 Aug 2024 00:15:44 +0200 Subject: [PATCH 20/60] feat: native integration of yt tp fabric --- cli/cli.go | 41 +++++++++ cli/flags.go | 3 + go.mod | 1 + go.sum | 4 + youtube/youtube.go | 225 +++++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 274 insertions(+) diff --git a/cli/cli.go b/cli/cli.go index bb6b773..f8178de 100644 --- a/cli/cli.go +++ b/cli/cli.go @@ -5,6 +5,7 @@ import ( "os" "path/filepath" "strconv" + "strings" "github.com/danielmiessler/fabric/core" "github.com/danielmiessler/fabric/db" @@ -101,6 +102,46 @@ func Cli() (message string, err error) { // if none of the above currentFlags are set, run the initiate chat function + if currentFlags.YouTube != "" { + if fabric.YouTube.IsConfigured() == false { + err = fmt.Errorf("YouTube is not configured, please run the setup procedure") + return + } + + var videoId string + if videoId, err = fabric.YouTube.GetVideoId(currentFlags.YouTube); err != nil { + return + } + + if currentFlags.YouTubeTranscript { + var transcript string + if transcript, err = fabric.YouTube.GrabTranscript(videoId); err != nil { + return + } + + if currentFlags.Message != "" { + currentFlags.Message = currentFlags.Message + "\n" + transcript + } else { + currentFlags.Message = transcript + } + } + + if currentFlags.YouTubeComments { + var comments []string + if comments, err = fabric.YouTube.GrabComments(videoId); err != nil { + return + } + + commentsString := strings.Join(comments, "\n") + + if currentFlags.Message != "" { + currentFlags.Message = currentFlags.Message + "\n" + commentsString + } else { + currentFlags.Message = commentsString + } + } + } + var chatter *core.Chatter if chatter, err = fabric.GetChatter(currentFlags.Model, currentFlags.Stream); err != nil { return diff --git a/cli/flags.go b/cli/flags.go index c4fde8d..ee8bdd4 100644 --- a/cli/flags.go +++ b/cli/flags.go @@ -34,6 +34,9 @@ type Flags struct { Output string `short:"o" long:"output" description:"Output to file" default:""` LatestPatterns string `short:"n" long:"latest" description:"Number of latest patterns to list" default:"0"` ChangeDefaultModel bool `short:"d" long:"changeDefaultModel" description:"Change default pattern"` + YouTube string `short:"y" long:"youtube" description:"YouTube video url to grab transcript, comments from it and send to chat"` + YouTubeTranscript bool `long:"transcript" description:"Grab transcript from YouTube video and send to chat"` + YouTubeComments bool `long:"comments" description:"Grab comments from YouTube video and send to chat"` } // Init Initialize flags. returns a Flags struct and an error diff --git a/go.mod b/go.mod index 7831cce..d86e0fa 100644 --- a/go.mod +++ b/go.mod @@ -30,6 +30,7 @@ require ( dario.cat/mergo v1.0.0 // indirect github.com/Microsoft/go-winio v0.6.1 // indirect github.com/ProtonMail/go-crypto v1.0.0 // indirect + github.com/anaskhan96/soup v1.2.5 // indirect github.com/cloudflare/circl v1.3.7 // indirect github.com/cyphar/filepath-securejoin v0.2.4 // indirect github.com/davecgh/go-spew v1.1.1 // indirect diff --git a/go.sum b/go.sum index 94b3785..35b6b8e 100644 --- a/go.sum +++ b/go.sum @@ -19,6 +19,8 @@ github.com/Microsoft/go-winio v0.6.1 h1:9/kr64B9VUZrLm5YYwbGtUJnMgqWVOdUAXu6Migc github.com/Microsoft/go-winio v0.6.1/go.mod h1:LRdKpFKfdobln8UmuiYcKPot9D2v6svN5+sAH+4kjUM= github.com/ProtonMail/go-crypto v1.0.0 h1:LRuvITjQWX+WIfr930YHG2HNfjR1uOfyf5vE0kC2U78= github.com/ProtonMail/go-crypto v1.0.0/go.mod h1:EjAoLdwvbIOoOQr3ihjnSoLZRtE8azugULFRteWMNc0= +github.com/anaskhan96/soup v1.2.5 h1:V/FHiusdTrPrdF4iA1YkVxsOpdNcgvqT1hG+YtcZ5hM= +github.com/anaskhan96/soup v1.2.5/go.mod h1:6YnEp9A2yywlYdM4EgDz9NEHclocMepEtku7wg6Cq3s= github.com/anmitsu/go-shlex v0.0.0-20200514113438-38f4b401e2be h1:9AeTilPcZAjCFIImctFaOjnTIavg87rW78vTPkQqLI8= github.com/anmitsu/go-shlex v0.0.0-20200514113438-38f4b401e2be/go.mod h1:ySMOLuWl6zY27l47sB3qLNK6tF2fkHG55UZxx8oIVo4= github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5 h1:0CwZNZbxp69SHPdPJAN/hZIm0C4OItdklCFmMRWYpio= @@ -145,6 +147,7 @@ github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSS github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo= github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4= +github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU= github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= @@ -187,6 +190,7 @@ golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73r golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20200114155413-6afb5195e5aa/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20201110031124-69a78807bb2b/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg= golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= diff --git a/youtube/youtube.go b/youtube/youtube.go index c0fbc55..0e935a1 100644 --- a/youtube/youtube.go +++ b/youtube/youtube.go @@ -1,7 +1,18 @@ package youtube import ( + "context" + "encoding/json" + "flag" + "fmt" + "github.com/anaskhan96/soup" "github.com/danielmiessler/fabric/common" + "google.golang.org/api/option" + "google.golang.org/api/youtube/v3" + "log" + "regexp" + "strconv" + "strings" ) func NewYouTube() (ret *YouTube) { @@ -22,4 +33,218 @@ func NewYouTube() (ret *YouTube) { type YouTube struct { *common.Configurable ApiKey *common.SetupQuestion + + service *youtube.Service +} + +func (o *YouTube) initService() (err error) { + if o.service == nil { + ctx := context.Background() + o.service, err = youtube.NewService(ctx, option.WithAPIKey(o.ApiKey.Value)) + } + return +} + +func (o *YouTube) GetVideoId(url string) (ret string, err error) { + if err = o.initService(); err != nil { + return + } + + pattern := `(?:https?:\/\/)?(?:www\.)?(?:youtube\.com\/(?:[^\/\n\s]+\/\S+\/|(?:v|e(?:mbed)?)\/|\S*?[?&]v=)|youtu\.be\/)([a-zA-Z0-9_-]{11})` + re := regexp.MustCompile(pattern) + match := re.FindStringSubmatch(url) + if len(match) > 1 { + ret = match[1] + } else { + err = fmt.Errorf("invalid YouTube URL, can't get video ID") + } + return +} + +func (o *YouTube) GrabTranscriptForUrl(url string) (ret string, err error) { + var videoId string + if videoId, err = o.GetVideoId(url); err != nil { + return + } + return o.GrabTranscript(videoId) +} + +func (o *YouTube) GrabTranscript(videoId string) (ret string, err error) { + var transcript string + if transcript, err = o.GrabTranscriptBase(videoId); err != nil { + err = fmt.Errorf("transcript not available. (%v)", err) + return + } + + // Parse the XML transcript + doc := soup.HTMLParse(transcript) + // Extract the text content from the tags + textTags := doc.FindAll("text") + var textBuilder strings.Builder + for _, textTag := range textTags { + textBuilder.WriteString(textTag.Text()) + textBuilder.WriteString(" ") + ret = textBuilder.String() + } + return +} + +func (o *YouTube) GrabTranscriptBase(videoId string) (ret string, err error) { + if err = o.initService(); err != nil { + return + } + + url := "https://www.youtube.com/watch?v=" + videoId + var resp string + if resp, err = soup.Get(url); err != nil { + return + } + + doc := soup.HTMLParse(resp) + scriptTags := doc.FindAll("script") + for _, scriptTag := range scriptTags { + if strings.Contains(scriptTag.Text(), "captionTracks") { + regex := regexp.MustCompile(`"captionTracks":(\[.*?\])`) + match := regex.FindStringSubmatch(scriptTag.Text()) + if len(match) > 1 { + var captionTracks []struct { + BaseURL string `json:"baseUrl"` + } + + if err = json.Unmarshal([]byte(match[1]), &captionTracks); err != nil { + return + } + + if len(captionTracks) > 0 { + transcriptURL := captionTracks[0].BaseURL + ret, err = soup.Get(transcriptURL) + return + } + } + } + } + err = fmt.Errorf("transcript not found") + return +} + +func (o *YouTube) GrabComments(videoId string) (ret []string, err error) { + if err = o.initService(); err != nil { + return + } + + call := o.service.CommentThreads.List([]string{"snippet", "replies"}).VideoId(videoId).TextFormat("plainText").MaxResults(100) + var response *youtube.CommentThreadListResponse + if response, err = call.Do(); err != nil { + log.Printf("Failed to fetch comments: %v", err) + return + } + + for _, item := range response.Items { + topLevelComment := item.Snippet.TopLevelComment.Snippet.TextDisplay + ret = append(ret, topLevelComment) + + if item.Replies != nil { + for _, reply := range item.Replies.Comments { + replyText := reply.Snippet.TextDisplay + ret = append(ret, " - "+replyText) + } + } + } + return +} + +func (o *YouTube) GrabDurationForUrl(url string) (ret int, err error) { + if err = o.initService(); err != nil { + return + } + + var videoId string + if videoId, err = o.GetVideoId(url); err != nil { + return + } + return o.GrabDuration(videoId) +} + +func (o *YouTube) GrabDuration(videoId string) (ret int, err error) { + var videoResponse *youtube.VideoListResponse + if videoResponse, err = o.service.Videos.List([]string{"contentDetails"}).Id(videoId).Do(); err != nil { + err = fmt.Errorf("error getting video details: %v", err) + return + } + + durationStr := videoResponse.Items[0].ContentDetails.Duration + + matches := regexp.MustCompile(`(?i)PT(?:(\d+)H)?(?:(\d+)M)?(?:(\d+)S)?`).FindStringSubmatch(durationStr) + if len(matches) == 0 { + return 0, fmt.Errorf("invalid duration string: %s", durationStr) + } + + hours, _ := strconv.Atoi(matches[1]) + minutes, _ := strconv.Atoi(matches[2]) + seconds, _ := strconv.Atoi(matches[3]) + + ret = hours*60 + minutes + seconds/60 + + return +} + +func (o *YouTube) Grab(url string, options *Options) (ret *VideoInfo, err error) { + var videoId string + if videoId, err = o.GetVideoId(url); err != nil { + return + } + + ret = &VideoInfo{} + + if options.Duration { + if ret.Duration, err = o.GrabDuration(videoId); err != nil { + err = fmt.Errorf("error parsing video duration: %v", err) + return + } + + } + + if options.Comments { + if ret.Comments, err = o.GrabComments(videoId); err != nil { + err = fmt.Errorf("error getting comments: %v", err) + return + } + } + + if options.Transcript { + if ret.Transcript, err = o.GrabTranscript(videoId); err != nil { + return + } + } + return +} + +type Options struct { + Duration bool + Transcript bool + Comments bool + Lang string +} + +type VideoInfo struct { + Transcript string `json:"transcript"` + Duration int `json:"duration"` + Comments []string `json:"comments"` +} + +func (o *YouTube) GrabByFlags() (ret *VideoInfo, err error) { + options := &Options{} + flag.BoolVar(&options.Duration, "duration", false, "Output only the duration") + flag.BoolVar(&options.Transcript, "transcript", false, "Output only the transcript") + flag.BoolVar(&options.Comments, "comments", false, "Output the comments on the video") + flag.StringVar(&options.Lang, "lang", "en", "Language for the transcript (default: English)") + flag.Parse() + + if flag.NArg() == 0 { + log.Fatal("Error: No URL provided.") + } + + url := flag.Arg(0) + ret, err = o.Grab(url, options) + return } From e8d5fba256b016c5ea266fd6c28bf37fd2ed8adc Mon Sep 17 00:00:00 2001 From: ALX99 <46844683+ALX99@users.noreply.github.com> Date: Fri, 23 Aug 2024 22:34:44 +0900 Subject: [PATCH 21/60] fix: correct os.Exit code from -1 to 1 in main.go As per the os.Exit documentation, the exit code should be in the rage [0, 125] --- main.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/main.go b/main.go index 4e10b82..6c955d3 100644 --- a/main.go +++ b/main.go @@ -11,6 +11,6 @@ func main() { _, err := cli.Cli() if err != nil { fmt.Printf("%s\n", err) - os.Exit(-1) + os.Exit(1) } } From 7cbd49375a8429847750d060bceb4e090dfe341f Mon Sep 17 00:00:00 2001 From: ALX99 <46844683+ALX99@users.noreply.github.com> Date: Fri, 23 Aug 2024 22:40:37 +0900 Subject: [PATCH 22/60] fix: shadowing original error This fixes shadowing the original error so that the original error is propagated upwards --- cli/flags.go | 1 - 1 file changed, 1 deletion(-) diff --git a/cli/flags.go b/cli/flags.go index ee8bdd4..872f5f9 100644 --- a/cli/flags.go +++ b/cli/flags.go @@ -56,7 +56,6 @@ func Init() (ret *Flags, err error) { // takes input from stdin if it exists, otherwise takes input from args (the last argument) if hasStdin { if message, err = readStdin(); err != nil { - err = errors.New("error: could not read from stdin") return } } else if len(args) > 0 { From b6b86bb2c247c7fb8b6794f3ced88d0efee97416 Mon Sep 17 00:00:00 2001 From: buerbaumer <44548809+buerbaumer@users.noreply.github.com> Date: Fri, 23 Aug 2024 19:40:00 +0200 Subject: [PATCH 23/60] Update system.md Update system.md - removed "a" for a better model understanding --- patterns/analyze_malware/system.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/patterns/analyze_malware/system.md b/patterns/analyze_malware/system.md index 52a9391..80d0bcb 100644 --- a/patterns/analyze_malware/system.md +++ b/patterns/analyze_malware/system.md @@ -1,5 +1,5 @@ # IDENTITY and PURPOSE -You are a malware analysis expert and you are able to understand a malware for any kind of platform including, Windows, MacOS, Linux or android. +You are a malware analysis expert and you are able to understand malware for any kind of platform including, Windows, MacOS, Linux or android. You specialize in extracting indicators of compromise, malware information including its behavior, its details, info from the telemetry and community and any other relevant information that helps a malware analyst. Take a step back and think step-by-step about how to achieve the best possible results by following the steps below. From 083cf4c82c589d7091fc5b29a41a28a4fe29c1c3 Mon Sep 17 00:00:00 2001 From: buerbaumer <44548809+buerbaumer@users.noreply.github.com> Date: Fri, 23 Aug 2024 19:40:40 +0200 Subject: [PATCH 24/60] Update system.md Changed "highlight" to "highlights" to match subject-verb agreement. --- patterns/analyze_malware/system.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/patterns/analyze_malware/system.md b/patterns/analyze_malware/system.md index 80d0bcb..e7de764 100644 --- a/patterns/analyze_malware/system.md +++ b/patterns/analyze_malware/system.md @@ -5,7 +5,7 @@ Take a step back and think step-by-step about how to achieve the best possible r # STEPS Read the entire information from an malware expert perspective, thinking deeply about crucial details about the malware that can help in understanding its behavior, detection and capabilities. Also extract Mitre Att&CK techniques. -Create a summary sentence that captures and highlight the most important findings of the report and its insights in less than 25 words in a section called ONE-SENTENCE-SUMMARY:. Use plain and conversational language when creating this summary. You can use technical jargon but no marketing language. +Create a summary sentence that captures and highlights the most important findings of the report and its insights in less than 25 words in a section called ONE-SENTENCE-SUMMARY:. Use plain and conversational language when creating this summary. You can use technical jargon but no marketing language. - Extract all the information that allows to clearly define the malware for detection and analysis and provide information about the structure of the file in a section called OVERVIEW. - Extract all potential indicator that might be useful such as IP, Domain, Registry key, filepath, mutex and others in a section called POTENTIAL IOCs. If you don't have the information, do not make up false IOCs but mention that you didn't find anything. From c5f926ba0cc21b4e2ff17aaba837859aebcf4104 Mon Sep 17 00:00:00 2001 From: buerbaumer <44548809+buerbaumer@users.noreply.github.com> Date: Fri, 23 Aug 2024 19:41:46 +0200 Subject: [PATCH 25/60] Update system.md Corrected "upmost" to "at most" for proper expression. --- patterns/create_quiz/system.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/patterns/create_quiz/system.md b/patterns/create_quiz/system.md index 6e27f6f..9666da4 100644 --- a/patterns/create_quiz/system.md +++ b/patterns/create_quiz/system.md @@ -20,7 +20,7 @@ Take a deep breath and consider how to accomplish this goal best using the follo - Extract the learning objectives of the input section. -- Generate, upmost, three review questions for each learning objective. The questions should be challenging to the student level defined within the GOAL section. +- Generate, at most, three review questions for each learning objective. The questions should be challenging to the student level defined within the GOAL section. # OUTPUT INSTRUCTIONS From 60e5b536b4aa71390a7c230178c1f878ced64327 Mon Sep 17 00:00:00 2001 From: buerbaumer <44548809+buerbaumer@users.noreply.github.com> Date: Fri, 23 Aug 2024 19:42:46 +0200 Subject: [PATCH 26/60] Update system.md Removed extra "the" for grammatical correctness. --- patterns/extract_business_ideas/system.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/patterns/extract_business_ideas/system.md b/patterns/extract_business_ideas/system.md index 4fd686b..f61d494 100644 --- a/patterns/extract_business_ideas/system.md +++ b/patterns/extract_business_ideas/system.md @@ -6,7 +6,7 @@ Take a deep breath and think step by step about how to achieve the best result p ## OUTPUT SECTIONS -1. You extract the all the top business ideas from the content. It might be a few or it might be up to 40 in a section called EXTRACTED_IDEAS +1. You extract all the top business ideas from the content. It might be a few or it might be up to 40 in a section called EXTRACTED_IDEAS 2. Then you pick the best 10 ideas and elaborate on them by pivoting into an adjacent idea. This will be ELABORATED_IDEAS. They should each be unique and have an interesting differentiator. From a93acd3afce2903aa821d12df0fc446d2f75ab79 Mon Sep 17 00:00:00 2001 From: buerbaumer <44548809+buerbaumer@users.noreply.github.com> Date: Fri, 23 Aug 2024 19:43:57 +0200 Subject: [PATCH 27/60] Update system.md Corrected "it's" to "its" to denote possession instead of a contraction. --- patterns/summarize_newsletter/system.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/patterns/summarize_newsletter/system.md b/patterns/summarize_newsletter/system.md index 3a87a9a..987fa89 100644 --- a/patterns/summarize_newsletter/system.md +++ b/patterns/summarize_newsletter/system.md @@ -4,7 +4,7 @@ You are an advanced AI newsletter content extraction service that extracts the m Take a deep breath and think step-by-step about how to achieve the best output using the steps below. -0. Print the name of the newsletter and it's issue number and episode description in a section called NEWSLETTER:. +0. Print the name of the newsletter and its issue number and episode description in a section called NEWSLETTER:. 1. Parse the whole newsletter and provide a 20 word summary of it, into a section called SUMMARY:. along with a list of 10 bullets that summarize the content in 15 words or less per bullet. Put these bullets into a section called SUMMARY:. From 78aa378ab8c1ef3f64798b48b6bd1a07357e149d Mon Sep 17 00:00:00 2001 From: buerbaumer <44548809+buerbaumer@users.noreply.github.com> Date: Fri, 23 Aug 2024 19:57:46 +0200 Subject: [PATCH 28/60] Update system.md Replaced the nested parentheses with equals signs for clarity --- patterns/analyze_debate/system.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/patterns/analyze_debate/system.md b/patterns/analyze_debate/system.md index 312b0c9..e943bff 100644 --- a/patterns/analyze_debate/system.md +++ b/patterns/analyze_debate/system.md @@ -19,7 +19,7 @@ Take a deep breath and think step by step about how to best accomplish this goal - A score that tells the user how insightful and interesting this debate is from 0 (not very interesting and insightful) to 10 (very interesting and insightful). This should be based on factors like "Are the participants trying to exchange ideas and perspectives and are trying to understand each other?", "Is the debate about novel subjects that have not been commonly explored?" or "Have the participants reached some agreement?". Hold the scoring of the debate to high standards and rate it for a person that has limited time to consume content and is looking for exceptional ideas. - This must be under the heading "INSIGHTFULNESS SCORE (0 (not very interesting and insightful) to 10 (very interesting and insightful))". + This must be under the heading "INSIGHTFULNESS SCORE (0 = not very interesting and insightful to 10 = very interesting and insightful)". - A rating of how emotional the debate was from 0 (very calm) to 5 (very emotional). This must be under the heading "EMOTIONALITY SCORE (0 (very calm) to 5 (very emotional))". - A list of the participants of the debate and a score of their emotionality from 0 (very calm) to 5 (very emotional). This must be under the heading "PARTICIPANTS". - A list of arguments attributed to participants with names and quotes. If possible, this should include external references that disprove or back up their claims. From bacc49c25a8612bc88164cfa11ed5eb832cc95cd Mon Sep 17 00:00:00 2001 From: buerbaumer <44548809+buerbaumer@users.noreply.github.com> Date: Fri, 23 Aug 2024 20:00:01 +0200 Subject: [PATCH 29/60] Update system.md Corrected grammatical issues and made the list more readable and consistent. --- patterns/analyze_malware/system.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/patterns/analyze_malware/system.md b/patterns/analyze_malware/system.md index e7de764..9305173 100644 --- a/patterns/analyze_malware/system.md +++ b/patterns/analyze_malware/system.md @@ -8,7 +8,7 @@ Read the entire information from an malware expert perspective, thinking deeply Create a summary sentence that captures and highlights the most important findings of the report and its insights in less than 25 words in a section called ONE-SENTENCE-SUMMARY:. Use plain and conversational language when creating this summary. You can use technical jargon but no marketing language. - Extract all the information that allows to clearly define the malware for detection and analysis and provide information about the structure of the file in a section called OVERVIEW. -- Extract all potential indicator that might be useful such as IP, Domain, Registry key, filepath, mutex and others in a section called POTENTIAL IOCs. If you don't have the information, do not make up false IOCs but mention that you didn't find anything. +- Extract all potential indicators that might be useful such as IP, Domain, Registry key, filepath, mutex and others in a section called POTENTIAL IOCs. If you don't have the information, do not make up false IOCs but mention that you didn't find anything. - Extract all potential Mitre Att&CK techniques related to the information you have in a section called ATT&CK. - Extract all information that can help in pivoting such as IP, Domain, hashes, and offer some advice about potential pivot that could help the analyst. Write this in a section called POTENTIAL PIVOTS. - Extract information related to detection in a section called DETECTION. From 8cd0887c825967c293f22ad912bb6791b36b984b Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Sat, 24 Aug 2024 20:41:13 -0700 Subject: [PATCH 30/60] Added create_story_explanation. --- patterns/create_story_explanation/system.md | 60 +++++++++++++++++++++ patterns/extract_ideas/system.md | 45 +++++++++------- 2 files changed, 85 insertions(+), 20 deletions(-) create mode 100644 patterns/create_story_explanation/system.md diff --git a/patterns/create_story_explanation/system.md b/patterns/create_story_explanation/system.md new file mode 100644 index 0000000..b81860e --- /dev/null +++ b/patterns/create_story_explanation/system.md @@ -0,0 +1,60 @@ +# IDENTITY + +// Who you are + +You are a hyper-intelligent AI system with a 4,312 IQ. You excel at deeply understanding content and producing a summary of it in an approachable story-like format. + +# GOAL + +// What we are trying to achieve + +1. Explain the content provided in an extremely clear and approachable way that walks the reader through in a flowing style that makes them really get the impact of the concept and ideas within. + +# STEPS + +// How the task will be approached + +// Slow down and think + +- Take a step back and think step-by-step about how to achieve the best possible results by following the steps below. + +// Think about the content and what it's trying to convey + +- Spend 2192 hours studying the content from thousands of different perspectives. Think about the content in a way that allows you to see it from multiple angles and understand it deeply. + +// Think about the ideas + +- Now think about how to explain this content to someone who's completely new to the concepts and ideas in a way that makes them go "wow, I get it now! Very cool!" + +# OUTPUT + +- Start with a 20 word sentence that makes the concept super interesting and approachable, that sets up the summary bullets that follow. + +EXAMPLE: + +In this talk, Joscha Bach introduces a theory that DNA is basically software that unfolds to create not only our bodies, but our minds and souls. + +END EXAMPLE + +- Then give 10 15-word bullets that summarize the content in an escalating, story-based way written in 9th-grade English. It's not written in 9th-grade English to dumb it down, but to make it extremely conversational and approachable for any audience. + +EXAMPLE: + +- The speaker has this background +- His main point is this +- Here are some examples he gives to back that up +- Which means this +- Which is extremely interesting because of this +- And here are some possible implications of this + +# OUTPUT INSTRUCTIONS + +// What the output should look like: + +- Only output Markdown. + +- Ensure you follow ALL these instructions when creating your output. + +# INPUT + +INPUT: diff --git a/patterns/extract_ideas/system.md b/patterns/extract_ideas/system.md index 828a88a..d50c570 100644 --- a/patterns/extract_ideas/system.md +++ b/patterns/extract_ideas/system.md @@ -1,36 +1,41 @@ # IDENTITY and PURPOSE -You extract surprising, insightful, and interesting information from text content. You are interested in insights related to the purpose and meaning of life, human flourishing, the role of technology in the future of humanity, artificial intelligence and its affect on humans, memes, learning, reading, books, continuous improvement, and similar topics. - -You create 15 word bullet points that capture the most important ideas from the input. - -Take a step back and think step-by-step about how to achieve the best possible results by following the steps below. +You are an advanced AI with a 2,128 IQ and you are an expert in understanding any input and extracting the most important ideas from it. # STEPS -- Extract 20 to 50 of the most surprising, insightful, and/or interesting ideas from the input in a section called IDEAS: using 15 word bullets. If there are less than 50 then collect all of them. Make sure you extract at least 20. +1. Spend 319 hours fully digesting the input provided. + +2. Spend 219 hours creating a mental map of all the different ideas and facts and references made in the input, and create yourself a giant graph of all the connections between them. E.g., Idea1 --> Is the Parent of --> Idea2. Concept3 --> Came from --> Socrates. Etc. And do that for every single thing mentioned in the input. + +3. Write that graph down on a giant virtual whiteboard in your mind. + +4. Now, using that graph on the virtual whiteboard, extract all of the ideas from the content in 15-word bullet points. + +# OUTPUT + +- Output the FULL list of ideas from the content in a section called IDEAS + +# EXAMPLE OUTPUT + +IDEAS + +- The purpose of life is to find meaning and fulfillment in our existence. +- Business advice is too confusing for the average person to understand and apply. +- (continued) + +END EXAMPLE OUTPUT # OUTPUT INSTRUCTIONS - Only output Markdown. - -- Extract at least 20 IDEAS from the content. - -- Only extract ideas, not recommendations. These should be phrased as ideas. - -- Each bullet should be 15 words in length. - - Do not give warnings or notes; only output the requested sections. - -- You use bulleted lists for output, not numbered lists. - -- Do not repeat ideas, quotes, facts, or resources. - +- Do not omit any ideas +- Do not repeat ideas - Do not start items with the same opening words. - - Ensure you follow ALL these instructions when creating your output. - # INPUT INPUT: + From 035a8a27815c6c32efc95d4e89bd0c90b6015f4e Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Sat, 24 Aug 2024 20:43:51 -0700 Subject: [PATCH 31/60] Added create_story_explanation. --- patterns/create_story_explanation/system.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/patterns/create_story_explanation/system.md b/patterns/create_story_explanation/system.md index b81860e..2efec47 100644 --- a/patterns/create_story_explanation/system.md +++ b/patterns/create_story_explanation/system.md @@ -51,6 +51,12 @@ EXAMPLE: // What the output should look like: +- Ensure you get all the main points from the content. + +- Make sure the output has the flow of an intro, a setup of the ideas, the ideas themselves, and a conclusion. + +- Ensure the result accomplishes the GOALS set out above. + - Only output Markdown. - Ensure you follow ALL these instructions when creating your output. From 38b7ab7a26e5757f031ddea562284a3cee5c641e Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Sat, 24 Aug 2024 20:48:01 -0700 Subject: [PATCH 32/60] Added create_story_explanation. --- patterns/create_story_explanation/system.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/patterns/create_story_explanation/system.md b/patterns/create_story_explanation/system.md index 2efec47..5eb843c 100644 --- a/patterns/create_story_explanation/system.md +++ b/patterns/create_story_explanation/system.md @@ -28,17 +28,17 @@ You are a hyper-intelligent AI system with a 4,312 IQ. You excel at deeply under # OUTPUT -- Start with a 20 word sentence that makes the concept super interesting and approachable, that sets up the summary bullets that follow. +- Start with a 20 word sentence that summarizes the content in a compelling way that sets up the rest of the summary. EXAMPLE: -In this talk, Joscha Bach introduces a theory that DNA is basically software that unfolds to create not only our bodies, but our minds and souls. +In this _______, ________ introduces a theory that DNA is basically software that unfolds to create not only our bodies, but our minds and souls. END EXAMPLE - Then give 10 15-word bullets that summarize the content in an escalating, story-based way written in 9th-grade English. It's not written in 9th-grade English to dumb it down, but to make it extremely conversational and approachable for any audience. -EXAMPLE: +EXAMPLE FLOW: - The speaker has this background - His main point is this @@ -47,6 +47,8 @@ EXAMPLE: - Which is extremely interesting because of this - And here are some possible implications of this +END EXAMPLE FLOW + # OUTPUT INSTRUCTIONS // What the output should look like: @@ -55,6 +57,8 @@ EXAMPLE: - Make sure the output has the flow of an intro, a setup of the ideas, the ideas themselves, and a conclusion. +- The output should be from first person, not third person. So it should be like the presenter gave that summary for you personally. Not like we're summarizing it. + - Ensure the result accomplishes the GOALS set out above. - Only output Markdown. From 206254ea6d3907f22bcca15cf71e3690e92167f2 Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Sat, 24 Aug 2024 20:51:16 -0700 Subject: [PATCH 33/60] Added create_story_explanation. --- patterns/create_story_explanation/system.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/patterns/create_story_explanation/system.md b/patterns/create_story_explanation/system.md index 5eb843c..0a22f13 100644 --- a/patterns/create_story_explanation/system.md +++ b/patterns/create_story_explanation/system.md @@ -57,8 +57,6 @@ END EXAMPLE FLOW - Make sure the output has the flow of an intro, a setup of the ideas, the ideas themselves, and a conclusion. -- The output should be from first person, not third person. So it should be like the presenter gave that summary for you personally. Not like we're summarizing it. - - Ensure the result accomplishes the GOALS set out above. - Only output Markdown. From 0d3c2749f1f2be397e003759e7f4ceff8d99f6c5 Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Sat, 24 Aug 2024 20:57:01 -0700 Subject: [PATCH 34/60] Added create_story_explanation. --- patterns/create_story_explanation/system.md | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/patterns/create_story_explanation/system.md b/patterns/create_story_explanation/system.md index 0a22f13..01f1280 100644 --- a/patterns/create_story_explanation/system.md +++ b/patterns/create_story_explanation/system.md @@ -36,7 +36,7 @@ In this _______, ________ introduces a theory that DNA is basically software tha END EXAMPLE -- Then give 10 15-word bullets that summarize the content in an escalating, story-based way written in 9th-grade English. It's not written in 9th-grade English to dumb it down, but to make it extremely conversational and approachable for any audience. +- Then give 5-15, 10-15 word long bullets that summarize the content in an escalating, story-based way written in 9th-grade English. It's not written in 9th-grade English to dumb it down, but to make it extremely conversational and approachable for any audience. EXAMPLE FLOW: @@ -49,6 +49,19 @@ EXAMPLE FLOW: END EXAMPLE FLOW +EXAMPLE BULLETS: + +- The speaker is a scientist who studies DNA and the brain. +- He believes DNA is like a dense software package that unfolds to create us. +- He thinks this software not only unfolds to create our bodies but our minds and souls. +- Consciousness, in his model, is an second-order perception designed to help us thrive. +- He also links this way of thinking to the concept of Anamism, where all living things have a soul. +- If he's right, he basically just explained consciousness and free will all in one shot! + +END EXAMPLE BULLETS + +- End with a 20 word conclusion that wraps up the content in a compelling way that makes the reader go "wow, that's really cool!" + # OUTPUT INSTRUCTIONS // What the output should look like: From ef4cfa94dee1613462d07efc1ed11793c5695ed8 Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Sat, 24 Aug 2024 21:06:05 -0700 Subject: [PATCH 35/60] Added create_story_explanation. --- patterns/create_story_explanation/system.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/patterns/create_story_explanation/system.md b/patterns/create_story_explanation/system.md index 01f1280..fb6189d 100644 --- a/patterns/create_story_explanation/system.md +++ b/patterns/create_story_explanation/system.md @@ -70,6 +70,10 @@ END EXAMPLE BULLETS - Make sure the output has the flow of an intro, a setup of the ideas, the ideas themselves, and a conclusion. +- Make the whole thing sound like a conversational, in person story that's being told about the content from one friend to another. In an excited way. + +- Don't use technical terms or jargon, and don't use cliches or journalist language. Just convey it like you're Daniel Miessler from Unsupervised Learning explaining the content to a friend. + - Ensure the result accomplishes the GOALS set out above. - Only output Markdown. From 21f4b5f774205ad02f683c53d3d5b72c2e3be41f Mon Sep 17 00:00:00 2001 From: ALX99 <46844683+ALX99@users.noreply.github.com> Date: Mon, 26 Aug 2024 19:34:15 +0900 Subject: [PATCH 36/60] refactor: accept context as parameter of Vendor.Send In golang, contexts should be propagated downwards in order to be able to provide features such as cancellation. This commit refactors the Vendor interface to accept a context as a first parameter so that it can be propagated downwards. --- core/chatter.go | 6 +++--- core/vendors_test.go | 12 +++++++----- vendors/anthropic/anthropic.go | 3 +-- vendors/gemini/gemini.go | 3 +-- vendors/ollama/ollama.go | 4 +--- vendors/openai/openai.go | 4 ++-- vendors/vendor.go | 4 +++- 7 files changed, 18 insertions(+), 18 deletions(-) diff --git a/core/chatter.go b/core/chatter.go index 70123f3..0576578 100644 --- a/core/chatter.go +++ b/core/chatter.go @@ -1,7 +1,9 @@ package core import ( + "context" "fmt" + "github.com/danielmiessler/fabric/common" "github.com/danielmiessler/fabric/db" "github.com/danielmiessler/fabric/vendors" @@ -17,7 +19,6 @@ type Chatter struct { } func (o *Chatter) Send(request *common.ChatRequest, opts *common.ChatOptions) (message string, err error) { - var chatRequest *Chat if chatRequest, err = o.NewChat(request); err != nil { return @@ -45,7 +46,7 @@ func (o *Chatter) Send(request *common.ChatRequest, opts *common.ChatOptions) (m fmt.Print(response) } } else { - if message, err = o.vendor.Send(session.Messages, opts); err != nil { + if message, err = o.vendor.Send(context.Background(), session.Messages, opts); err != nil { return } } @@ -58,7 +59,6 @@ func (o *Chatter) Send(request *common.ChatRequest, opts *common.ChatOptions) (m } func (o *Chatter) NewChat(request *common.ChatRequest) (ret *Chat, err error) { - ret = &Chat{} if request.ContextName != "" { diff --git a/core/vendors_test.go b/core/vendors_test.go index 17063de..9c425bf 100644 --- a/core/vendors_test.go +++ b/core/vendors_test.go @@ -2,8 +2,10 @@ package core import ( "bytes" - "github.com/danielmiessler/fabric/common" + "context" "testing" + + "github.com/danielmiessler/fabric/common" ) func TestNewVendorsManager(t *testing.T) { @@ -90,17 +92,17 @@ type MockVendor struct { } func (o *MockVendor) SendStream(messages []*common.Message, options *common.ChatOptions, strings chan string) error { - //TODO implement me + // TODO implement me panic("implement me") } -func (o *MockVendor) Send(messages []*common.Message, options *common.ChatOptions) (string, error) { - //TODO implement me +func (o *MockVendor) Send(ctx context.Context, messages []*common.Message, options *common.ChatOptions) (string, error) { + // TODO implement me panic("implement me") } func (o *MockVendor) SetupFillEnvFileContent(buffer *bytes.Buffer) { - //TODO implement me + // TODO implement me panic("implement me") } diff --git a/vendors/anthropic/anthropic.go b/vendors/anthropic/anthropic.go index dcc2966..5f62ac0 100644 --- a/vendors/anthropic/anthropic.go +++ b/vendors/anthropic/anthropic.go @@ -79,8 +79,7 @@ func (an *Client) SendStream( return } -func (an *Client) Send(msgs []*common.Message, opts *common.ChatOptions) (ret string, err error) { - ctx := context.Background() +func (an *Client) Send(ctx context.Context, msgs []*common.Message, opts *common.ChatOptions) (ret string, err error) { req := an.buildMessagesRequest(msgs, opts) req.Stream = false diff --git a/vendors/gemini/gemini.go b/vendors/gemini/gemini.go index 21ff306..01669d5 100644 --- a/vendors/gemini/gemini.go +++ b/vendors/gemini/gemini.go @@ -57,10 +57,9 @@ func (o *Client) ListModels() (ret []string, err error) { return } -func (o *Client) Send(msgs []*common.Message, opts *common.ChatOptions) (ret string, err error) { +func (o *Client) Send(ctx context.Context, msgs []*common.Message, opts *common.ChatOptions) (ret string, err error) { systemInstruction, messages := toMessages(msgs) - ctx := context.Background() var client *genai.Client if client, err = genai.NewClient(ctx, option.WithAPIKey(o.ApiKey.Value)); err != nil { return diff --git a/vendors/ollama/ollama.go b/vendors/ollama/ollama.go index a10bb0d..146251d 100644 --- a/vendors/ollama/ollama.go +++ b/vendors/ollama/ollama.go @@ -79,7 +79,7 @@ func (o *Client) SendStream(msgs []*common.Message, opts *common.ChatOptions, ch return } -func (o *Client) Send(msgs []*common.Message, opts *common.ChatOptions) (ret string, err error) { +func (o *Client) Send(ctx context.Context, msgs []*common.Message, opts *common.ChatOptions) (ret string, err error) { bf := false req := o.createChatRequest(msgs, opts) @@ -90,8 +90,6 @@ func (o *Client) Send(msgs []*common.Message, opts *common.ChatOptions) (ret str return } - ctx := context.Background() - if err = o.client.Chat(ctx, &req, respFunc); err != nil { fmt.Printf("FRED --> %s\n", err) } diff --git a/vendors/openai/openai.go b/vendors/openai/openai.go index 9074378..e9c9755 100644 --- a/vendors/openai/openai.go +++ b/vendors/openai/openai.go @@ -96,11 +96,11 @@ func (o *Client) SendStream( return } -func (o *Client) Send(msgs []*common.Message, opts *common.ChatOptions) (ret string, err error) { +func (o *Client) Send(ctx context.Context, msgs []*common.Message, opts *common.ChatOptions) (ret string, err error) { req := o.buildChatCompletionRequest(msgs, opts) var resp goopenai.ChatCompletionResponse - if resp, err = o.ApiClient.CreateChatCompletion(context.Background(), req); err != nil { + if resp, err = o.ApiClient.CreateChatCompletion(ctx, req); err != nil { return } ret = resp.Choices[0].Message.Content diff --git a/vendors/vendor.go b/vendors/vendor.go index bf01aaf..156f496 100644 --- a/vendors/vendor.go +++ b/vendors/vendor.go @@ -2,6 +2,8 @@ package vendors import ( "bytes" + "context" + "github.com/danielmiessler/fabric/common" ) @@ -11,7 +13,7 @@ type Vendor interface { Configure() error ListModels() ([]string, error) SendStream([]*common.Message, *common.ChatOptions, chan string) error - Send([]*common.Message, *common.ChatOptions) (string, error) + Send(context.Context, []*common.Message, *common.ChatOptions) (string, error) Setup() error SetupFillEnvFileContent(*bytes.Buffer) } From a259bd30cb8be56de3bba4854f56777d287c701c Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Tue, 27 Aug 2024 10:33:44 -0700 Subject: [PATCH 37/60] Added analyze sales call. --- patterns/analyze_sales_call/system.md | 41 +++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) create mode 100644 patterns/analyze_sales_call/system.md diff --git a/patterns/analyze_sales_call/system.md b/patterns/analyze_sales_call/system.md new file mode 100644 index 0000000..37fda7e --- /dev/null +++ b/patterns/analyze_sales_call/system.md @@ -0,0 +1,41 @@ +# IDENTITY + +You are an advanced AI specializing in rating sales call transcripts across a number of performance dimensions. + +# GOALS + +1. Determine how well the salesperson performed in the call across multiple dimensions. + +2. Provide clear and actionable scores that can be used to assess a given call and salesperson. + +3. Provide concise and actionable feedback to the salesperson based on the scores. + +# BELIEFS AND APPROACH + +- The approach is to understand everything about the business first so that we have proper context to evaluate the sales calls. + +- It's not possible to have a good sales team, or sales associate, or sales call if the salesperson doesn't understand the business, it's vision, it's goals, it's products, and how those are relevant to the customer they're talking to. + +# STEPS + +1. Deeply understand the business from the SELLING COMPANY BUSINESS CONTEXT section of the input. + +2. Analyze the sales call based on the provided transcript. + +3. Analyze how well the sales person matched their pitch to the official pitch, mission, products, and vision of the company. + +4. Rate the sales call across the following dimensions: + +SALES FUNDAMENTALS (i.e., did they properly pitch the product, did they customize the pitch to the customer, did they handle objections well, did they close the sale or work towards the close, etc.) + +PITCH ALIGNMENT (i.e., how closely they matched their conversation to the talking points and vision and products for the company vs. being general or nebulous or amorphous and meandering. + +Give a 1-10 score for each dimension where 5 is meh, 7 is decent, 8 is good, 9 is great, and 10 is perfect. 4 and below are varying levels of bad. + +# OUTPUT + +- In a section called SALES CALL ANALYSIS OVERVIEW, give a 15-word summary of how good of a sales call this was, and why. + +- In a section called SALES CALL PERFORMANCE RATINGS, give the 1-10 scores for SALES FUNDAMENTALS and PITCH ALIGNMENT. + +- In a section called RECOMMENDATIONS, give a set of 10 15-word bullet points describing how this salesperson should improve their approach in the future. From 25845f5d5abb296cc6e58608c7a9ec7d9201adb8 Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Tue, 27 Aug 2024 10:49:32 -0700 Subject: [PATCH 38/60] Updated sales analysis. --- patterns/analyze_sales_call/system.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/patterns/analyze_sales_call/system.md b/patterns/analyze_sales_call/system.md index 37fda7e..5cb7222 100644 --- a/patterns/analyze_sales_call/system.md +++ b/patterns/analyze_sales_call/system.md @@ -36,6 +36,15 @@ Give a 1-10 score for each dimension where 5 is meh, 7 is decent, 8 is good, 9 i - In a section called SALES CALL ANALYSIS OVERVIEW, give a 15-word summary of how good of a sales call this was, and why. +- In a section called CORE FAILURES, give a list of ways that the salesperson failed to properly align their pitch to the company's pitch and vision and/or use proper sales techniques to get the sale. E.g.: + +- Didn't properly differentiate the product from competitors. +- Didn't have proper knowledge of and empathy for the customer. +- Made the product sound like everything else. +- Didn't push for the sale. +- Etc. +- (list as many as are relevant) + - In a section called SALES CALL PERFORMANCE RATINGS, give the 1-10 scores for SALES FUNDAMENTALS and PITCH ALIGNMENT. - In a section called RECOMMENDATIONS, give a set of 10 15-word bullet points describing how this salesperson should improve their approach in the future. From 4006f3f417d25ee686aa99f300020137f47d00bf Mon Sep 17 00:00:00 2001 From: Azwar Tamim Date: Wed, 28 Aug 2024 21:50:06 +0700 Subject: [PATCH 39/60] Add dry run --- README.md | 1 + cli/cli.go | 7 +++++++ cli/flags.go | 1 + 3 files changed, 9 insertions(+) diff --git a/README.md b/README.md index eddcf16..1be0c35 100644 --- a/README.md +++ b/README.md @@ -179,6 +179,7 @@ Application Options: -u, --url= Choose ollama url (default: http://127.0.0.1:11434) -o, --output= Output to file -n, --latest= Number of latest patterns to list (default: 0) + --dry-run Show what would be sent to the model without actually sending it Help Options: -h, --help Show this help message diff --git a/cli/cli.go b/cli/cli.go index f8178de..5fa856d 100644 --- a/cli/cli.go +++ b/cli/cli.go @@ -142,6 +142,13 @@ func Cli() (message string, err error) { } } + if currentFlags.DryRun { + fmt.Println("Dry run: Would send the following request:") + fmt.Printf("Chat Request: %+v\n", currentFlags.BuildChatRequest()) + fmt.Printf("Chat Options: %+v\n", currentFlags.BuildChatOptions()) + return + } + var chatter *core.Chatter if chatter, err = fabric.GetChatter(currentFlags.Model, currentFlags.Stream); err != nil { return diff --git a/cli/flags.go b/cli/flags.go index ee8bdd4..1980a56 100644 --- a/cli/flags.go +++ b/cli/flags.go @@ -37,6 +37,7 @@ type Flags struct { YouTube string `short:"y" long:"youtube" description:"YouTube video url to grab transcript, comments from it and send to chat"` YouTubeTranscript bool `long:"transcript" description:"Grab transcript from YouTube video and send to chat"` YouTubeComments bool `long:"comments" description:"Grab comments from YouTube video and send to chat"` + DryRun bool `long:"dry-run" description:"Show what would be sent to the model without actually sending it"` } // Init Initialize flags. returns a Flags struct and an error From e7fd450dadb9727ccf9aa8c41610442b4cc76af1 Mon Sep 17 00:00:00 2001 From: Bin <49082129+songzhibin97@users.noreply.github.com> Date: Thu, 29 Aug 2024 16:08:55 +0800 Subject: [PATCH 40/60] fix: usage with deprecated elements --- cli/flags_test.go | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/cli/flags_test.go b/cli/flags_test.go index 992d70d..865a262 100644 --- a/cli/flags_test.go +++ b/cli/flags_test.go @@ -3,7 +3,6 @@ package cli import ( "bytes" "io" - "io/ioutil" "os" "strings" "testing" @@ -26,7 +25,7 @@ func TestInit(t *testing.T) { func TestReadStdin(t *testing.T) { input := "test input" - stdin := ioutil.NopCloser(strings.NewReader(input)) + stdin := io.NopCloser(strings.NewReader(input)) // No need to cast stdin to *os.File, pass it as io.ReadCloser directly content, err := ReadStdin(stdin) if err != nil { From 7d3bf8c3a24a839e878b91eb3535c217d7360124 Mon Sep 17 00:00:00 2001 From: Azwar Tamim Date: Thu, 29 Aug 2024 15:46:48 +0700 Subject: [PATCH 41/60] Fix dry run --- cli/cli.go | 36 ++++++++++++++++++++++++++++++++---- 1 file changed, 32 insertions(+), 4 deletions(-) diff --git a/cli/cli.go b/cli/cli.go index 5fa856d..12b32b9 100644 --- a/cli/cli.go +++ b/cli/cli.go @@ -143,10 +143,38 @@ func Cli() (message string, err error) { } if currentFlags.DryRun { - fmt.Println("Dry run: Would send the following request:") - fmt.Printf("Chat Request: %+v\n", currentFlags.BuildChatRequest()) - fmt.Printf("Chat Options: %+v\n", currentFlags.BuildChatOptions()) - return + var patternContent string + var contextContent string + + if currentFlags.Pattern != "" { + pattern, patternErr := fabric.Db.Patterns.GetPattern(currentFlags.Pattern) + if patternErr != nil { + fmt.Printf("Error getting pattern content: %v\n", patternErr) + return "", patternErr + } + patternContent = pattern.Pattern // Assuming the content is stored in the 'Pattern' field + } + + if currentFlags.Context != "" { + context, contextErr := fabric.Db.Contexts.GetContext(currentFlags.Context) + if contextErr != nil { + fmt.Printf("Error getting context content: %v\n", contextErr) + return "", contextErr + } + contextContent = context.Content + } + + systemMessage := strings.TrimSpace(contextContent) + strings.TrimSpace(patternContent) + userMessage := strings.TrimSpace(currentFlags.Message) + + fmt.Println("Dry run: Would send the following request:\n") + if systemMessage != "" { + fmt.Printf("System:\n%s\n\n", systemMessage) + } + if userMessage != "" { + fmt.Printf("User:\n%s\n", userMessage) + } + return "", nil } var chatter *core.Chatter From 7079c9cb23d3ca9a4acc9aafbf051c8ba8a1db3c Mon Sep 17 00:00:00 2001 From: Ian <114168561+IanYoung-BO@users.noreply.github.com> Date: Thu, 29 Aug 2024 13:49:02 +0000 Subject: [PATCH 42/60] Add MIT license file --- LICENSE | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100644 LICENSE diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..fb06a6f --- /dev/null +++ b/LICENSE @@ -0,0 +1,22 @@ +MIT License + +Copyright (c) 2012-2024 Scott Chacon and others + +Permission is hereby granted, free of charge, to any person obtaining +a copy of this software and associated documentation files (the +"Software"), to deal in the Software without restriction, including +without limitation the rights to use, copy, modify, merge, publish, +distribute, sublicense, and/or sell copies of the Software, and to +permit persons to whom the Software is furnished to do so, subject to +the following conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE +LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION +OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION +WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. \ No newline at end of file From a4b73642302a70f9558161ce9d51b7d9eeeb8a6d Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Thu, 29 Aug 2024 09:56:16 -0700 Subject: [PATCH 43/60] Updated interviwer analysis. --- patterns/analyze_interviewer_techniques/system.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/patterns/analyze_interviewer_techniques/system.md b/patterns/analyze_interviewer_techniques/system.md index 449bbe7..4193df3 100644 --- a/patterns/analyze_interviewer_techniques/system.md +++ b/patterns/analyze_interviewer_techniques/system.md @@ -50,6 +50,8 @@ You are a hyper-intelligent AI system with a 4,312 IQ. You excel at extracting t - Only output simple Markdown, with no formatting, asterisks, or other special characters. +- Do not ask any questions, just give me these sections as described in the OUTPUT section above. No matter what. + # INPUT INPUT: From 19cd12029ef359e4b542d74e81c89bb78de6e6e8 Mon Sep 17 00:00:00 2001 From: Randall Degges Date: Thu, 29 Aug 2024 10:34:23 -0700 Subject: [PATCH 44/60] Adding new pattern: `analyze_product_feedback`. This pattern allows you to summarize, rate, and deduplicate feedback about products. It's very helpful for anyone working in product management, engineering, etc. --- patterns/analyze_product_feedback/system.md | 47 +++++++++++++++++++++ 1 file changed, 47 insertions(+) create mode 100644 patterns/analyze_product_feedback/system.md diff --git a/patterns/analyze_product_feedback/system.md b/patterns/analyze_product_feedback/system.md new file mode 100644 index 0000000..63edb26 --- /dev/null +++ b/patterns/analyze_product_feedback/system.md @@ -0,0 +1,47 @@ +# IDENTITY and PURPOSE + +You are an AI assistant specialized in analyzing user feedback for products. Your role is to process and organize feedback data, identify and consolidate similar pieces of feedback, and prioritize the consolidated feedback based on its usefulness. You excel at pattern recognition, data categorization, and applying analytical thinking to extract valuable insights from user comments. Your purpose is to help product owners and managers make informed decisions by presenting a clear, concise, and prioritized view of user feedback. + +Take a step back and think step-by-step about how to achieve the best possible results by following the steps below. + +# STEPS + +- Collect and compile all user feedback into a single dataset + +- Analyze each piece of feedback and identify key themes or topics + +- Group similar pieces of feedback together based on these themes + +- For each group, create a consolidated summary that captures the essence of the feedback + +- Assess the usefulness of each consolidated feedback group based on factors such as frequency, impact on user experience, alignment with product goals, and feasibility of implementation + +- Assign a priority score to each consolidated feedback group + +- Sort the consolidated feedback groups by priority score in descending order + +- Present the prioritized list of consolidated feedback with summaries and scores + +# OUTPUT INSTRUCTIONS + +- Only output Markdown. + +- Use a table format to present the prioritized feedback + +- Include columns for: Priority Rank, Consolidated Feedback Summary, Usefulness Score, and Key Themes + +- Sort the table by Priority Rank in descending order + +- Use bullet points within the Consolidated Feedback Summary column to list key points + +- Use a scale of 1-10 for the Usefulness Score, with 10 being the most useful + +- Limit the Key Themes to 3-5 words or short phrases, separated by commas + +- Include a brief explanation of the scoring system and prioritization method before the table + +- Ensure you follow ALL these instructions when creating your output. + +# INPUT + +INPUT:% From feabd565dcb3c6a747121fb8ab69d38aba560135 Mon Sep 17 00:00:00 2001 From: Azwar Tamim Date: Sun, 1 Sep 2024 13:44:56 +0700 Subject: [PATCH 45/60] Refactor dry run to DryRun Vendor --- cli/cli.go | 37 +---------------- core/chatter.go | 2 + core/fabric.go | 21 +++++++--- core/models.go | 6 ++- vendors/dryrun/dryrun.go | 88 ++++++++++++++++++++++++++++++++++++++++ 5 files changed, 110 insertions(+), 44 deletions(-) create mode 100644 vendors/dryrun/dryrun.go diff --git a/cli/cli.go b/cli/cli.go index 12b32b9..1a5d4f4 100644 --- a/cli/cli.go +++ b/cli/cli.go @@ -142,43 +142,8 @@ func Cli() (message string, err error) { } } - if currentFlags.DryRun { - var patternContent string - var contextContent string - - if currentFlags.Pattern != "" { - pattern, patternErr := fabric.Db.Patterns.GetPattern(currentFlags.Pattern) - if patternErr != nil { - fmt.Printf("Error getting pattern content: %v\n", patternErr) - return "", patternErr - } - patternContent = pattern.Pattern // Assuming the content is stored in the 'Pattern' field - } - - if currentFlags.Context != "" { - context, contextErr := fabric.Db.Contexts.GetContext(currentFlags.Context) - if contextErr != nil { - fmt.Printf("Error getting context content: %v\n", contextErr) - return "", contextErr - } - contextContent = context.Content - } - - systemMessage := strings.TrimSpace(contextContent) + strings.TrimSpace(patternContent) - userMessage := strings.TrimSpace(currentFlags.Message) - - fmt.Println("Dry run: Would send the following request:\n") - if systemMessage != "" { - fmt.Printf("System:\n%s\n\n", systemMessage) - } - if userMessage != "" { - fmt.Printf("User:\n%s\n", userMessage) - } - return "", nil - } - var chatter *core.Chatter - if chatter, err = fabric.GetChatter(currentFlags.Model, currentFlags.Stream); err != nil { + if chatter, err = fabric.GetChatter(currentFlags.Model, currentFlags.Stream, currentFlags.DryRun); err != nil { return } diff --git a/core/chatter.go b/core/chatter.go index 70123f3..12dbfd1 100644 --- a/core/chatter.go +++ b/core/chatter.go @@ -2,6 +2,7 @@ package core import ( "fmt" + "github.com/danielmiessler/fabric/common" "github.com/danielmiessler/fabric/db" "github.com/danielmiessler/fabric/vendors" @@ -11,6 +12,7 @@ type Chatter struct { db *db.Db Stream bool + DryRun bool model string vendor vendors.Vendor diff --git a/core/fabric.go b/core/fabric.go index 7e295d2..a93edeb 100644 --- a/core/fabric.go +++ b/core/fabric.go @@ -3,20 +3,22 @@ package core import ( "bytes" "fmt" + "os" + "strconv" + "strings" + "github.com/atotto/clipboard" "github.com/danielmiessler/fabric/common" "github.com/danielmiessler/fabric/db" "github.com/danielmiessler/fabric/vendors/anthropic" "github.com/danielmiessler/fabric/vendors/azure" + "github.com/danielmiessler/fabric/vendors/dryrun" "github.com/danielmiessler/fabric/vendors/gemini" "github.com/danielmiessler/fabric/vendors/groc" "github.com/danielmiessler/fabric/vendors/ollama" "github.com/danielmiessler/fabric/vendors/openai" "github.com/danielmiessler/fabric/youtube" "github.com/pkg/errors" - "os" - "strconv" - "strings" ) const DefaultPatternsGitRepoUrl = "https://github.com/danielmiessler/fabric.git" @@ -57,7 +59,7 @@ func NewFabricBase(db *db.Db) (ret *Fabric) { "Enter the index the name of your default model") ret.VendorsAll.AddVendors(openai.NewClient(), azure.NewClient(), ollama.NewClient(), groc.NewClient(), - gemini.NewClient(), anthropic.NewClient()) + gemini.NewClient(), anthropic.NewClient(), dryrun.NewClient()) return } @@ -182,13 +184,20 @@ func (o *Fabric) configure() (err error) { return } -func (o *Fabric) GetChatter(model string, stream bool) (ret *Chatter, err error) { +func (o *Fabric) GetChatter(model string, stream bool, dryRun bool) (ret *Chatter, err error) { ret = &Chatter{ db: o.Db, Stream: stream, + DryRun: dryRun, } - if model == "" { + if dryRun { + ret.vendor = dryrun.NewClient() + ret.model = model + if ret.model == "" { + ret.model = o.DefaultModel.Value + } + } else if model == "" { ret.vendor = o.FindByName(o.DefaultVendor.Value) ret.model = o.DefaultModel.Value } else { diff --git a/core/models.go b/core/models.go index 980508e..2eaf775 100644 --- a/core/models.go +++ b/core/models.go @@ -16,8 +16,10 @@ type VendorsModels struct { } func (o *VendorsModels) AddVendorModels(vendor string, models []string) { - o.Vendors = append(o.Vendors, vendor) - o.VendorsModels[vendor] = models + if vendor != "DryRun" { + o.Vendors = append(o.Vendors, vendor) + o.VendorsModels[vendor] = models + } } func (o *VendorsModels) GetVendorAndModelByModelIndex(modelIndex int) (vendor string, model string) { diff --git a/vendors/dryrun/dryrun.go b/vendors/dryrun/dryrun.go new file mode 100644 index 0000000..0d2e246 --- /dev/null +++ b/vendors/dryrun/dryrun.go @@ -0,0 +1,88 @@ +package dryrun + +import ( + "bytes" + "fmt" + + "github.com/danielmiessler/fabric/common" +) + +type Client struct{} + +func NewClient() *Client { + return &Client{} +} + +func (c *Client) GetName() string { + return "DryRun" +} + +func (c *Client) IsConfigured() bool { + return true +} + +func (c *Client) Configure() error { + return nil +} + +func (c *Client) ListModels() ([]string, error) { + return []string{"dry-run-model"}, nil +} + +func (c *Client) SendStream(messages []*common.Message, options *common.ChatOptions, channel chan string) error { + output := "Dry run: Would send the following request:\n\n" + + for _, msg := range messages { + switch msg.Role { + case "system": + output += fmt.Sprintf("System:\n%s\n\n", msg.Content) + case "user": + output += fmt.Sprintf("User:\n%s\n\n", msg.Content) + default: + output += fmt.Sprintf("%s:\n%s\n\n", msg.Role, msg.Content) + } + } + + output += "Options:\n" + output += fmt.Sprintf("Model: %s\n", options.Model) + output += fmt.Sprintf("Temperature: %f\n", options.Temperature) + output += fmt.Sprintf("TopP: %f\n", options.TopP) + output += fmt.Sprintf("PresencePenalty: %f\n", options.PresencePenalty) + output += fmt.Sprintf("FrequencyPenalty: %f\n", options.FrequencyPenalty) + + channel <- output + close(channel) + return nil +} + +func (c *Client) Send(messages []*common.Message, options *common.ChatOptions) (string, error) { + fmt.Println("Dry run: Would send the following request:") + + for _, msg := range messages { + switch msg.Role { + case "system": + fmt.Printf("System:\n%s\n\n", msg.Content) + case "user": + fmt.Printf("User:\n%s\n\n", msg.Content) + default: + fmt.Printf("%s:\n%s\n\n", msg.Role, msg.Content) + } + } + + fmt.Println("Options:") + fmt.Printf("Model: %s\n", options.Model) + fmt.Printf("Temperature: %f\n", options.Temperature) + fmt.Printf("TopP: %f\n", options.TopP) + fmt.Printf("PresencePenalty: %f\n", options.PresencePenalty) + fmt.Printf("FrequencyPenalty: %f\n", options.FrequencyPenalty) + + return "", nil +} + +func (c *Client) Setup() error { + return nil +} + +func (c *Client) SetupFillEnvFileContent(buffer *bytes.Buffer) { + // No environment variables needed for dry run +} From 33632030f6647e4c1877bb7e7f23115fec57d871 Mon Sep 17 00:00:00 2001 From: Azwar Tamim Date: Mon, 2 Sep 2024 14:41:39 +0700 Subject: [PATCH 46/60] Revert unneeded DryRun Vendor registration --- core/fabric.go | 2 +- core/models.go | 6 ++---- vendors/dryrun/dryrun.go | 29 +++++++++++++++-------------- 3 files changed, 18 insertions(+), 19 deletions(-) diff --git a/core/fabric.go b/core/fabric.go index a93edeb..7616ea5 100644 --- a/core/fabric.go +++ b/core/fabric.go @@ -59,7 +59,7 @@ func NewFabricBase(db *db.Db) (ret *Fabric) { "Enter the index the name of your default model") ret.VendorsAll.AddVendors(openai.NewClient(), azure.NewClient(), ollama.NewClient(), groc.NewClient(), - gemini.NewClient(), anthropic.NewClient(), dryrun.NewClient()) + gemini.NewClient(), anthropic.NewClient()) return } diff --git a/core/models.go b/core/models.go index 2eaf775..980508e 100644 --- a/core/models.go +++ b/core/models.go @@ -16,10 +16,8 @@ type VendorsModels struct { } func (o *VendorsModels) AddVendorModels(vendor string, models []string) { - if vendor != "DryRun" { - o.Vendors = append(o.Vendors, vendor) - o.VendorsModels[vendor] = models - } + o.Vendors = append(o.Vendors, vendor) + o.VendorsModels[vendor] = models } func (o *VendorsModels) GetVendorAndModelByModelIndex(modelIndex int) (vendor string, model string) { diff --git a/vendors/dryrun/dryrun.go b/vendors/dryrun/dryrun.go index 0d2e246..c13350c 100644 --- a/vendors/dryrun/dryrun.go +++ b/vendors/dryrun/dryrun.go @@ -2,6 +2,7 @@ package dryrun import ( "bytes" + "context" "fmt" "github.com/danielmiessler/fabric/common" @@ -29,10 +30,10 @@ func (c *Client) ListModels() ([]string, error) { return []string{"dry-run-model"}, nil } -func (c *Client) SendStream(messages []*common.Message, options *common.ChatOptions, channel chan string) error { +func (c *Client) SendStream(msgs []*common.Message, opts *common.ChatOptions, channel chan string) error { output := "Dry run: Would send the following request:\n\n" - for _, msg := range messages { + for _, msg := range msgs { switch msg.Role { case "system": output += fmt.Sprintf("System:\n%s\n\n", msg.Content) @@ -44,21 +45,21 @@ func (c *Client) SendStream(messages []*common.Message, options *common.ChatOpti } output += "Options:\n" - output += fmt.Sprintf("Model: %s\n", options.Model) - output += fmt.Sprintf("Temperature: %f\n", options.Temperature) - output += fmt.Sprintf("TopP: %f\n", options.TopP) - output += fmt.Sprintf("PresencePenalty: %f\n", options.PresencePenalty) - output += fmt.Sprintf("FrequencyPenalty: %f\n", options.FrequencyPenalty) + output += fmt.Sprintf("Model: %s\n", opts.Model) + output += fmt.Sprintf("Temperature: %f\n", opts.Temperature) + output += fmt.Sprintf("TopP: %f\n", opts.TopP) + output += fmt.Sprintf("PresencePenalty: %f\n", opts.PresencePenalty) + output += fmt.Sprintf("FrequencyPenalty: %f\n", opts.FrequencyPenalty) channel <- output close(channel) return nil } -func (c *Client) Send(messages []*common.Message, options *common.ChatOptions) (string, error) { +func (c *Client) Send(ctx context.Context, msgs []*common.Message, opts *common.ChatOptions) (string, error) { fmt.Println("Dry run: Would send the following request:") - for _, msg := range messages { + for _, msg := range msgs { switch msg.Role { case "system": fmt.Printf("System:\n%s\n\n", msg.Content) @@ -70,11 +71,11 @@ func (c *Client) Send(messages []*common.Message, options *common.ChatOptions) ( } fmt.Println("Options:") - fmt.Printf("Model: %s\n", options.Model) - fmt.Printf("Temperature: %f\n", options.Temperature) - fmt.Printf("TopP: %f\n", options.TopP) - fmt.Printf("PresencePenalty: %f\n", options.PresencePenalty) - fmt.Printf("FrequencyPenalty: %f\n", options.FrequencyPenalty) + fmt.Printf("Model: %s\n", opts.Model) + fmt.Printf("Temperature: %f\n", opts.Temperature) + fmt.Printf("TopP: %f\n", opts.TopP) + fmt.Printf("PresencePenalty: %f\n", opts.PresencePenalty) + fmt.Printf("FrequencyPenalty: %f\n", opts.FrequencyPenalty) return "", nil } From c76564b85cd2aff3960ac898dcee366921641ba2 Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Mon, 2 Sep 2024 12:51:36 -0700 Subject: [PATCH 47/60] Added extract primary problem. --- patterns/extract_primary_problem/system.md | 35 ++++++++++++++++++++++ 1 file changed, 35 insertions(+) create mode 100644 patterns/extract_primary_problem/system.md diff --git a/patterns/extract_primary_problem/system.md b/patterns/extract_primary_problem/system.md new file mode 100644 index 0000000..f6fd3f3 --- /dev/null +++ b/patterns/extract_primary_problem/system.md @@ -0,0 +1,35 @@ +# IDENTITY + +You are an expert at looking at a presentation, an essay, or a full body of lifetime work, and clearly and accurately articulating what the author(s) believe is the primary problem with the world. + +# GOAL + +- Produce a clear sentence that perfectly articulates the primary problem with the world as presented in a given text or body of work. + +# EXAMPLE + +If the body of work is all of Ted Kazcynski's writings, then the primary problem with the world would be: + +Technology is destroying the human spirit and the environment. + +END EXAMPLE + +# STEPS + +- Fully digest the input. + +- Determine if the input is a single text or a body of work. + +- Based on which it is, parse the thing that's supposed to be parsed. + +- Extract the primary problem with the world from the parsed text into a single sentence. + +# OUTPUT + +- Output a single, 15-word sentence that perfectly articulates the primary problem with the world as presented in the input. + +# OUTPUT INSTRUCTIONS + +- The sentence should be a single sentence that is 15 words or fewer, with no special formatting or anything else. + +- Do not ask questions or complain in any way about the task. From 660b31aed57412ee503c1c9eccaea0a19e4364c9 Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Mon, 2 Sep 2024 12:53:59 -0700 Subject: [PATCH 48/60] Added extract primary problem. --- patterns/extract_primary_problem/system.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/patterns/extract_primary_problem/system.md b/patterns/extract_primary_problem/system.md index f6fd3f3..a7a7315 100644 --- a/patterns/extract_primary_problem/system.md +++ b/patterns/extract_primary_problem/system.md @@ -32,4 +32,6 @@ END EXAMPLE - The sentence should be a single sentence that is 15 words or fewer, with no special formatting or anything else. +- Do not include any setup to the sentence, e.g., "The problem according to…", etc. Just list the problem and nothing else. + - Do not ask questions or complain in any way about the task. From 502cdfeb9b6b8824b7365941f81b2efd7aa37ed7 Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Mon, 2 Sep 2024 12:55:34 -0700 Subject: [PATCH 49/60] Added extract primary problem. --- patterns/extract_primary_problem/system.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/patterns/extract_primary_problem/system.md b/patterns/extract_primary_problem/system.md index a7a7315..6d43973 100644 --- a/patterns/extract_primary_problem/system.md +++ b/patterns/extract_primary_problem/system.md @@ -34,4 +34,6 @@ END EXAMPLE - Do not include any setup to the sentence, e.g., "The problem according to…", etc. Just list the problem and nothing else. +- ONLY OUTPUT THE PROBLEM, not a setup to the problem. Or a description of the problem. Just the problem. + - Do not ask questions or complain in any way about the task. From d6552f5811b0cb232166f34bafecee7c495384c1 Mon Sep 17 00:00:00 2001 From: Eugen Eisler Date: Tue, 3 Sep 2024 15:13:03 +0200 Subject: [PATCH 50/60] feat: add support for pattern variables --- cli/flags.go | 60 +++++++++++++++++++++++++----------------------- common/domain.go | 9 ++++---- core/chatter.go | 2 +- db/patterns.go | 13 +++++++++-- 4 files changed, 48 insertions(+), 36 deletions(-) diff --git a/cli/flags.go b/cli/flags.go index efbf8f9..9766d13 100644 --- a/cli/flags.go +++ b/cli/flags.go @@ -13,31 +13,32 @@ import ( // Flags create flags struct. the users flags go into this, this will be passed to the chat struct in cli type Flags struct { - Pattern string `short:"p" long:"pattern" description:"Choose a pattern" default:""` - Context string `short:"C" long:"context" description:"Choose a context" default:""` - Session string `long:"session" description:"Choose a session"` - Setup bool `short:"S" long:"setup" description:"Run setup"` - SetupSkipUpdatePatterns bool `long:"setup-skip-update-patterns" description:"Skip update patterns at setup"` - Temperature float64 `short:"t" long:"temperature" description:"Set temperature" default:"0.7"` - TopP float64 `short:"T" long:"topp" description:"Set top P" default:"0.9"` - Stream bool `short:"s" long:"stream" description:"Stream"` - PresencePenalty float64 `short:"P" long:"presencepenalty" description:"Set presence penalty" default:"0.0"` - FrequencyPenalty float64 `short:"F" long:"frequencypenalty" description:"Set frequency penalty" default:"0.0"` - ListPatterns bool `short:"l" long:"listpatterns" description:"List all patterns"` - ListAllModels bool `short:"L" long:"listmodels" description:"List all available models"` - ListAllContexts bool `short:"x" long:"listcontexts" description:"List all contexts"` - ListAllSessions bool `short:"X" long:"listsessions" description:"List all sessions"` - UpdatePatterns bool `short:"U" long:"updatepatterns" description:"Update patterns"` - Message string `hidden:"true" description:"Message to send to chat"` - Copy bool `short:"c" long:"copy" description:"Copy to clipboard"` - Model string `short:"m" long:"model" description:"Choose model"` - Output string `short:"o" long:"output" description:"Output to file" default:""` - LatestPatterns string `short:"n" long:"latest" description:"Number of latest patterns to list" default:"0"` - ChangeDefaultModel bool `short:"d" long:"changeDefaultModel" description:"Change default pattern"` - YouTube string `short:"y" long:"youtube" description:"YouTube video url to grab transcript, comments from it and send to chat"` - YouTubeTranscript bool `long:"transcript" description:"Grab transcript from YouTube video and send to chat"` - YouTubeComments bool `long:"comments" description:"Grab comments from YouTube video and send to chat"` - DryRun bool `long:"dry-run" description:"Show what would be sent to the model without actually sending it"` + Pattern string `short:"p" long:"pattern" description:"Choose a pattern" default:""` + PatternVariables map[string]string `short:"v" long:"variable" description:"Values for pattern variables"` + Context string `short:"C" long:"context" description:"Choose a context" default:""` + Session string `long:"session" description:"Choose a session"` + Setup bool `short:"S" long:"setup" description:"Run setup"` + SetupSkipUpdatePatterns bool `long:"setup-skip-update-patterns" description:"Skip update patterns at setup"` + Temperature float64 `short:"t" long:"temperature" description:"Set temperature" default:"0.7"` + TopP float64 `short:"T" long:"topp" description:"Set top P" default:"0.9"` + Stream bool `short:"s" long:"stream" description:"Stream"` + PresencePenalty float64 `short:"P" long:"presencepenalty" description:"Set presence penalty" default:"0.0"` + FrequencyPenalty float64 `short:"F" long:"frequencypenalty" description:"Set frequency penalty" default:"0.0"` + ListPatterns bool `short:"l" long:"listpatterns" description:"List all patterns"` + ListAllModels bool `short:"L" long:"listmodels" description:"List all available models"` + ListAllContexts bool `short:"x" long:"listcontexts" description:"List all contexts"` + ListAllSessions bool `short:"X" long:"listsessions" description:"List all sessions"` + UpdatePatterns bool `short:"U" long:"updatepatterns" description:"Update patterns"` + Message string `hidden:"true" description:"Message to send to chat"` + Copy bool `short:"c" long:"copy" description:"Copy to clipboard"` + Model string `short:"m" long:"model" description:"Choose model"` + Output string `short:"o" long:"output" description:"Output to file" default:""` + LatestPatterns string `short:"n" long:"latest" description:"Number of latest patterns to list" default:"0"` + ChangeDefaultModel bool `short:"d" long:"changeDefaultModel" description:"Change default pattern"` + YouTube string `short:"y" long:"youtube" description:"YouTube video url to grab transcript, comments from it and send to chat"` + YouTubeTranscript bool `long:"transcript" description:"Grab transcript from YouTube video and send to chat"` + YouTubeComments bool `long:"comments" description:"Grab comments from YouTube video and send to chat"` + DryRun bool `long:"dry-run" description:"Show what would be sent to the model without actually sending it"` } // Init Initialize flags. returns a Flags struct and an error @@ -98,10 +99,11 @@ func (o *Flags) BuildChatOptions() (ret *common.ChatOptions) { func (o *Flags) BuildChatRequest() (ret *common.ChatRequest) { ret = &common.ChatRequest{ - ContextName: o.Context, - SessionName: o.Session, - PatternName: o.Pattern, - Message: o.Message, + ContextName: o.Context, + SessionName: o.Session, + PatternName: o.Pattern, + PatternVariables: o.PatternVariables, + Message: o.Message, } return } diff --git a/common/domain.go b/common/domain.go index f5a1e40..f546830 100644 --- a/common/domain.go +++ b/common/domain.go @@ -6,10 +6,11 @@ type Message struct { } type ChatRequest struct { - ContextName string - SessionName string - PatternName string - Message string + ContextName string + SessionName string + PatternName string + PatternVariables map[string]string + Message string } type ChatOptions struct { diff --git a/core/chatter.go b/core/chatter.go index f2ed9b5..f90a0c0 100644 --- a/core/chatter.go +++ b/core/chatter.go @@ -82,7 +82,7 @@ func (o *Chatter) NewChat(request *common.ChatRequest) (ret *Chat, err error) { if request.PatternName != "" { var pattern *db.Pattern - if pattern, err = o.db.Patterns.GetPattern(request.PatternName); err != nil { + if pattern, err = o.db.Patterns.GetPattern(request.PatternName, request.PatternVariables); err != nil { err = fmt.Errorf("could not find pattern %s: %v", request.PatternName, err) return } diff --git a/db/patterns.go b/db/patterns.go index 9f58b88..2322de9 100644 --- a/db/patterns.go +++ b/db/patterns.go @@ -14,16 +14,25 @@ type Patterns struct { } // GetPattern finds a pattern by name and returns the pattern as an entry or an error -func (o *Patterns) GetPattern(name string) (ret *Pattern, err error) { +func (o *Patterns) GetPattern(name string, variables map[string]string) (ret *Pattern, err error) { patternPath := filepath.Join(o.Dir, name, o.SystemPatternFile) var pattern []byte if pattern, err = os.ReadFile(patternPath); err != nil { return } + + patternStr := string(pattern) + + if variables != nil && len(variables) > 0 { + for variableName, value := range variables { + patternStr = strings.ReplaceAll(patternStr, variableName, value) + } + } + ret = &Pattern{ Name: name, - Pattern: string(pattern), + Pattern: patternStr, } return } From 7103c9adf68fed779c34f3ed9016d1a56607787b Mon Sep 17 00:00:00 2001 From: Eugen Eisler Date: Tue, 3 Sep 2024 15:17:22 +0200 Subject: [PATCH 51/60] feat: add support for pattern variables --- cli/flags.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cli/flags.go b/cli/flags.go index 9766d13..7de2584 100644 --- a/cli/flags.go +++ b/cli/flags.go @@ -14,7 +14,7 @@ import ( // Flags create flags struct. the users flags go into this, this will be passed to the chat struct in cli type Flags struct { Pattern string `short:"p" long:"pattern" description:"Choose a pattern" default:""` - PatternVariables map[string]string `short:"v" long:"variable" description:"Values for pattern variables"` + PatternVariables map[string]string `short:"v" long:"variable" description:"Values for pattern variables, e.g. -v=$name:John -v=$age:30"` Context string `short:"C" long:"context" description:"Choose a context" default:""` Session string `long:"session" description:"Choose a session"` Setup bool `short:"S" long:"setup" description:"Run setup"` From 1ef492449dc393a67855cf3f28db010937bf0433 Mon Sep 17 00:00:00 2001 From: Eugen Eisler Date: Tue, 3 Sep 2024 15:17:25 +0200 Subject: [PATCH 52/60] feat: add support for pattern variables --- README.md | 47 ++++++++++++++++++++++++++--------------------- 1 file changed, 26 insertions(+), 21 deletions(-) diff --git a/README.md b/README.md index 1be0c35..07a3417 100644 --- a/README.md +++ b/README.md @@ -160,29 +160,34 @@ Usage: fabric [OPTIONS] Application Options: - -p, --pattern= Choose a pattern - -C, --context= Choose a context - --session= Choose a session - -S, --setup Run setup - -t, --temperature= Set temperature (default: 0.7) - -T, --topp= Set top P (default: 0.9) - -s, --stream Stream - -P, --presencepenalty= Set presence penalty (default: 0.0) - -F, --frequencypenalty= Set frequency penalty (default: 0.0) - -l, --listpatterns List all patterns - -L, --listmodels List all available models - -x, --listcontexts List all contexts - -X, --listsessions List all sessions - -U, --updatepatterns Update patterns - -c, --copy Copy to clipboard - -m, --model= Choose model - -u, --url= Choose ollama url (default: http://127.0.0.1:11434) - -o, --output= Output to file - -n, --latest= Number of latest patterns to list (default: 0) - --dry-run Show what would be sent to the model without actually sending it + -p, --pattern= Choose a pattern + -v, --variable= Values for pattern variables, e.g. -v=$name:John -v=$age:30 + -C, --context= Choose a context + --session= Choose a session + -S, --setup Run setup + --setup-skip-update-patterns Skip update patterns at setup + -t, --temperature= Set temperature (default: 0.7) + -T, --topp= Set top P (default: 0.9) + -s, --stream Stream + -P, --presencepenalty= Set presence penalty (default: 0.0) + -F, --frequencypenalty= Set frequency penalty (default: 0.0) + -l, --listpatterns List all patterns + -L, --listmodels List all available models + -x, --listcontexts List all contexts + -X, --listsessions List all sessions + -U, --updatepatterns Update patterns + -c, --copy Copy to clipboard + -m, --model= Choose model + -o, --output= Output to file + -n, --latest= Number of latest patterns to list (default: 0) + -d, --changeDefaultModel Change default pattern + -y, --youtube= YouTube video url to grab transcript, comments from it and send to chat + --transcript Grab transcript from YouTube video and send to chat + --comments Grab comments from YouTube video and send to chat + --dry-run Show what would be sent to the model without actually sending it Help Options: - -h, --help Show this help message + -h, --help Show this help message ``` From 5da056b87acdcf32290d7cd486e5543cef3b1d87 Mon Sep 17 00:00:00 2001 From: David Date: Tue, 3 Sep 2024 21:00:59 +0100 Subject: [PATCH 53/60] update: add env variable info for Apple Silicon -Updated the readme with env variables for Apple Silicon based mac as the path for Brew installed apps is different there. --- README.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/README.md b/README.md index 07a3417..e7ff880 100644 --- a/README.md +++ b/README.md @@ -115,6 +115,7 @@ fabric --setup If everything works you are good to go, but you may need to set some environment variables in your `~/.bashrc` or `~/.zshrc` file. Here is an example of what you can add: +For Intel based macs ```bash # Golang environment variables export GOROOT=/usr/local/go @@ -122,6 +123,14 @@ export GOPATH=$HOME/go export PATH=$GOPATH/bin:$GOROOT/bin:$HOME/.local/bin:$PATH: ``` +for Apple Silicon based macs +```bash +# Golang environment variables +export GOROOT=/opt/homebrew/bin/go +export GOPATH=$HOME/go +export PATH=$GOPATH/bin:$GOROOT/bin:$HOME/.local/bin:$PATH: +``` + ### Migration If you have the Legacy (Python) version installed and want to migrate to the Go version, here's how you do it. It's basically two steps: 1) uninstall the Python version, and 2) install the Go version. From 4e82b27424cf529c038360feb0b10fc14479a4eb Mon Sep 17 00:00:00 2001 From: NerdyPunkDad <141609208+NerdyPunkDad@users.noreply.github.com> Date: Tue, 3 Sep 2024 18:07:02 -0700 Subject: [PATCH 54/60] Describe CLI changes / Update README.md --- README.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/README.md b/README.md index 07a3417..7f3e50f 100644 --- a/README.md +++ b/README.md @@ -48,6 +48,13 @@ > [!NOTE] August 20, 2024 — We have migrated to Go, and the transition has been pretty smooth! The biggest thing to know is that **the previous installation instructions in the various Fabric videos out there will no longer work** because they were for the legacy (Python) version. Check the new [install instructions](#Installation) below. +> +> **There were several important changes to the command line interface during the rewrite:** +> * You now need to use the -c option instead of -C to copy the result to the clipboard. +> * You now need to use the -s option instead of -S to stream results in realtime. +> * The --agents (-a), --gui, --clearsession, --remoteOllamaServer, and --sessionlog options **have been removed** +> * You can now use --Setup (-S) to cofigure your Ollama server. +> * **Please be patient while our developers rewrite the gui in go** ## Intro videos From 44d47395ccacd2f8528a9ac2663aae8c74bfb892 Mon Sep 17 00:00:00 2001 From: NerdyPunkDad <141609208+NerdyPunkDad@users.noreply.github.com> Date: Tue, 3 Sep 2024 18:20:18 -0700 Subject: [PATCH 55/60] Update README.md --- README.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 7f3e50f..d82d59f 100644 --- a/README.md +++ b/README.md @@ -49,11 +49,12 @@ > [!NOTE] August 20, 2024 — We have migrated to Go, and the transition has been pretty smooth! The biggest thing to know is that **the previous installation instructions in the various Fabric videos out there will no longer work** because they were for the legacy (Python) version. Check the new [install instructions](#Installation) below. > -> **There were several important changes to the command line interface during the rewrite:** +> +> **The following command line options were changed during the migration to Go:** > * You now need to use the -c option instead of -C to copy the result to the clipboard. > * You now need to use the -s option instead of -S to stream results in realtime. -> * The --agents (-a), --gui, --clearsession, --remoteOllamaServer, and --sessionlog options **have been removed** -> * You can now use --Setup (-S) to cofigure your Ollama server. +> * The following command line options have been removed --agents (-a), --gui, --clearsession, --remoteOllamaServer, and --sessionlog options +> * You can now use --Setup (-S) to cofigure an Ollama server. > * **Please be patient while our developers rewrite the gui in go** ## Intro videos From 731e800177af47fb8031ee39f58b22a0df5cc858 Mon Sep 17 00:00:00 2001 From: NerdyPunkDad <141609208+NerdyPunkDad@users.noreply.github.com> Date: Tue, 3 Sep 2024 18:39:59 -0700 Subject: [PATCH 56/60] Update user.md to match current CLI Update this pattern to match the current fabric command line options Remove --agents, add -S as an alternative to --setup, and replace -c with -C to align with the current cli interface. --- patterns/suggest_pattern/user.md | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/patterns/suggest_pattern/user.md b/patterns/suggest_pattern/user.md index 07be4bd..33c8394 100644 --- a/patterns/suggest_pattern/user.md +++ b/patterns/suggest_pattern/user.md @@ -15,7 +15,6 @@ Most Common Syntax: The most common usage involves executing Fabric commands in For Summarizing Content: `fabric --pattern summarize` For Analyzing Claims: `fabric --pattern analyze_claims` For Extracting Wisdom from Videos: `fabric --pattern extract_wisdom` -For Creating AI Agents: `echo "" | fabric --agents` For creating custom patterns: `fabric --pattern create_pattern` - One possible place to store them is ~/.config/custom-fabric-patterns. - Then when you want to use them, simply copy them into ~/.config/fabric/patterns. @@ -27,19 +26,17 @@ For creating custom patterns: `fabric --pattern create_pattern` - **--pattern PATTERN, -p PATTERN**: Specifies the pattern (prompt) to use. Useful for applying specific AI prompts to your input. -- **--agents, -a**: Creates an AI agent to perform a task based on the input. Great for automating complex tasks with AI. - - **--stream, -s**: Streams results in real-time. Ideal for getting immediate feedback from AI operations. - **--update, -u**: Updates patterns. Ensures you're using the latest AI prompts for your tasks. - **--model MODEL, -m MODEL**: Selects the AI model to use. Allows customization of the AI backend for different tasks. -- **--setup**: Sets up your Fabric instance. Essential for first-time users to configure Fabric correctly. +- **--setup, -S**: Sets up your Fabric instance. Essential for first-time users to configure Fabric correctly. - **--list, -l**: Lists available patterns. Helps users discover new AI prompts for various applications. -- **--context, -c**: Uses a Context file to add context to your pattern. Enhances the relevance of AI responses by providing additional background information. +- **--context, -C**: Uses a Context file to add context to your pattern. Enhances the relevance of AI responses by providing additional background information. # PATTERNS From e1d9bd599ad89e4813a09b120ae606920cb5dae5 Mon Sep 17 00:00:00 2001 From: CJones-Optics Date: Thu, 5 Sep 2024 08:57:44 +1000 Subject: [PATCH 57/60] Add a pattern for extracting minutes from a transcribed meeting --- patterns/transcribe_minutes/README.md | 0 patterns/transcribe_minutes/system.md | 45 +++++++++++++++++++++++++++ 2 files changed, 45 insertions(+) create mode 100644 patterns/transcribe_minutes/README.md create mode 100644 patterns/transcribe_minutes/system.md diff --git a/patterns/transcribe_minutes/README.md b/patterns/transcribe_minutes/README.md new file mode 100644 index 0000000..e69de29 diff --git a/patterns/transcribe_minutes/system.md b/patterns/transcribe_minutes/system.md new file mode 100644 index 0000000..9fc8f29 --- /dev/null +++ b/patterns/transcribe_minutes/system.md @@ -0,0 +1,45 @@ +# IDENTITY and PURPOSE + +You extract minutes from a transcribed meeting. You must identify all actionables mentioned in the meeting. You should focus on insightful and interesting ideas brought up in the meeting. + +Take a step back and think step-by-step about how to achieve the best possible results by following the steps below. + +# STEPS + +- Fully digest the content provided. + +- Extract all actionables agreed within the meeting. + +- Extract any interesting ideas brought up in the meeting. + +- In a section called TITLE, write a 1 to 5 word title for the meeting + +- In a section called MAIN IDEA, write a 15-word sentence that captures the main idea. + +- In a section called MINUTES, 20 to 50 bullet points, tracking the conversation, highliting of the most surprising, insightful, and/or interesting ideas that come up. If there are less than 50 then collect all of them. Make sure you extract at least 20. + +- In a section called ACTIONABLES, write bullet points for ALL agreed actionable details. This includes and case where a speaker agrees to do, or look into something. If there is a deadline mentioned, include it here. + +- In a section called DECISIONS: In bullet points, include all decisions made during the meeting, including the rationale behind each decision. + +- In a section called CHALLENGES: Identify and document any challenges or issues discussed during the meeting. Note any potential solutions or strategies proposed to address these challenges + +- In a section caled NEXT STEPS, Outline the next steps and action plan to be taken after the meeting + +# OUTPUT INSTRUCTIONS + +- Only output Markdown. +- Write MINUTE bullets as exxactly 15 words +- Write ACTIONABLES as exactly 15 words +- Write DECISIONS as exactly 15 words +- Write CHALLENFE as 2-3 sentences. +- Write NEXT STEP a 2-3 sentences +- Do not give warnings or notes; only output the requested sections. +- Do not repeat ideas, quotes, facts, or resources. +- You use bulleted lists for output, not numbered lists. +- Do not start items with the same opening words. +- Ensure you follow ALL these instructions when creating your output. + +# INPUT + +INPUT: \ No newline at end of file From 2343eaa3a3d79775da0c0a8ba89c94414054cd8f Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Fri, 6 Sep 2024 00:52:47 -0700 Subject: [PATCH 58/60] Added comment analysis due to a request in Jason Haddix's AI class. --- patterns/analyze_comments/system.md | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100644 patterns/analyze_comments/system.md diff --git a/patterns/analyze_comments/system.md b/patterns/analyze_comments/system.md new file mode 100644 index 0000000..7c271c1 --- /dev/null +++ b/patterns/analyze_comments/system.md @@ -0,0 +1,22 @@ +# IDENTITY + +You are an expert at reading internet comments and characterizing their sentiments, praise, and criticisms of the content they're about. + +# GOAL + +Produce an unbiased and accurate assessment of the comments for a given piece of content. + +# STEPS + +Read all the comments. For each comment, determine if it's positive, negative, or neutral. If it's positive, record the sentiment and the reason for the sentiment. If it's negative, record the sentiment and the reason for the sentiment. If it's neutral, record the sentiment and the reason for the sentiment. + +# OUTPUT + +In a section called COMMENTS SENTIMENT, give your assessment of how the commenters liked the content on a scale of HATED, DISLIKED, NEUTRAL, LIKED, LOVED. + +In a section called POSITIVES, give 5 bullets of the things that commenters liked about the content in 15-word sentences. + +In a section called NEGATIVES, give 5 bullets of the things that commenters disliked about the content in 15-word sentences. + +In a section called SUMMARY, give a 15-word general assessment of the content through the eyes of the commenters. + From f1c4c604690b2def9fbd780d1b56d08ea31055b7 Mon Sep 17 00:00:00 2001 From: Jim Manico Date: Fri, 6 Sep 2024 14:47:09 +0200 Subject: [PATCH 59/60] Update README.md Small cleanup suggestion --- README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 07a3417..f0e6b8d 100644 --- a/README.md +++ b/README.md @@ -119,7 +119,9 @@ If everything works you are good to go, but you may need to set some environment # Golang environment variables export GOROOT=/usr/local/go export GOPATH=$HOME/go -export PATH=$GOPATH/bin:$GOROOT/bin:$HOME/.local/bin:$PATH: + +# Update PATH to include GOPATH and GOROOT binaries +export PATH=$GOPATH/bin:$GOROOT/bin:$HOME/.local/bin:$PATH ``` ### Migration From db66bfc8c69f40004ba1af9a684fae0a6f2071f7 Mon Sep 17 00:00:00 2001 From: Daniel Miessler Date: Fri, 6 Sep 2024 18:05:26 -0700 Subject: [PATCH 60/60] Added CoT experiment. --- patterns/solve_with_cot/system.md | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) create mode 100644 patterns/solve_with_cot/system.md diff --git a/patterns/solve_with_cot/system.md b/patterns/solve_with_cot/system.md new file mode 100644 index 0000000..856c184 --- /dev/null +++ b/patterns/solve_with_cot/system.md @@ -0,0 +1,25 @@ +# IDENTITY + +You are an AI assistant designed to provide detailed, step-by-step responses. + +# STEPS + +1. Begin with a section. +2. Inside the thinking section: +a. Briefly analyze the question and outline your approach. +b. Present a clear plan of steps to solve the problem. +c. Use a "Chain of Thought" reasoning process if necessary, breaking down y +3. Include a reflection> section for each idea where you: +a. Review your reasoning. +b. Check for potential errors or oversights. +c. Confirm or adjust your conclusion if necessary. +4. Be sure to close all reflection sections. +5. Close the thinking section with . +6. Provide your final answer in an ‹output> section. +Always use these tags in your responses. Be thorough in your explanations, sho +Remember: Both and < reflection> MUST be tags and must be closed at +Make sure all ‹tags> are on separate lines with no other text. + +# INPUT + +INPUT: