Generate Knowledge Graphs with LLMs
Overview
The original post.
The model being used is mistralai/Mistral-7B-Instruct-v0.2.
The UI I'm using for inference is oobabooga's text-generation-webui.
Input
I am using the first page of The Hobbit as a test input.
1In a hole in the ground there lived a hobbit. Not a nasty, dirty, wet hole, filled with the
2ends of worms and an oozy smell, nor yet a dry, bare, sandy hole with nothing in it to sit down
3on or to eat: it was a hobbithole, and that means comfort.
4It had a perfectly round door like a porthole, painted green, with a shiny yellow brass knob
5in the exact middle. The door opened on to a tubeshaped hall like a tunnel: a very comfortable
6tunnel without smoke, with panelled walls, and floors tiled and carpeted, provided with polished
7chairs, and lots and lots of pegs for hats and coats the hobbit was fond of visitors. The tunnel
8wound on and on, going fairly but not quite straight into the side of the hill The Hill, as all the
9people for many miles round called it and many little round doors opened out of it, first on
10one side and then on another. No going upstairs for the hobbit: bedrooms, bathrooms, cellars,
11pantries (lots of these), wardrobes (he had whole rooms devoted to clothes), kitchens,
12diningrooms, all were on the same floor, and indeed on the same passage. The best rooms
13were all on the lefthand side (going in), for these were the only ones to have windows,
14deepset round windows looking over his garden and meadows beyond, sloping down to the
15river.
16This hobbit was a very welltodo hobbit, and his name was Baggins. The Bagginses had
17lived in the neighbourhood of The Hill for time out of mind, and people considered them very
18respectable, not only because most of them were rich, but also because they never had any
19adventures or did anything unexpected: you could tell what a Baggins would say on any
20question without the bother of asking him. This is a story of how a Baggins had an adventure,
21found himself doing and saying things altogether unexpected. He may have lost the
22neighbours' respect, but he gainedwell, you will see whether he gained anything in the end.
23The mother of our particular hobbit … what is a hobbit? I suppose hobbits need some
24description nowadays, since they have become rare and shy of the Big People, as they call us.
25They are (or were) a little people, about half our height, and smaller than the bearded Dwarves.
26Hobbits have no beards. There is little or no magic about them, except the ordinary everyday
27sort which helps them to disappear quietly and quickly when large stupid folk like you and me
28come blundering along, making a noise like elephants which they can hear a mile off. They are
29inclined to be at in the stomach; they dress in bright colours (chiefly green and yellow); wear
30no shoes, because their feet grow natural leathery soles and thick warm brown hair like the
31stuff on their heads (which is curly); have long clever brown fingers, goodnatured faces, and
32laugh deep fruity laughs (especially after dinner, which they have twice a day when they can
33get it). Now you know enough to go on with. As I was saying, the mother of this hobbit of
34Bilbo Baggins, that is was the fabulous Belladonna Took, one of the three remarkable
35daughters of the Old Took, head of the hobbits who lived across The Water, the small river that
36ran at the foot of The Hill. It was often said (in other families) that long ago one of the Took
37ancestors must have taken a fairy wife. That was, of course, absurd, but certainly there was
38still something not entirely hobbitlike about them, and once in a while members of the
39Tookclan would go and have adventures. They discreetly disappeared, and the family hushed
40it up; but the fact remained that the Tooks were not as respectable as the Bagginses, though
41they were undoubtedly richer. Not that Belladonna Took ever had any adventures after she
42became Mrs. Bungo Baggins. Bungo, that was Bilbo's father, built the most luxurious
43hobbithole for her (and partly with her money) that was to be found either under The Hill or
44over The Hill or across The Water, and there they remained to the end of their days. Still it is
45probable that Bilbo, her only son, although he looked and behaved exactly like a second edition
46of his solid and comfortable father, got something a bit queer in his makeup from the Took
47side, something that only waited for a chance to come out. The chance never arrived, until
48Bilbo Baggins was grown up, being about fifty years old or so, and living in the beautiful
49hobbithole built by his father, which I have just described for you, until he had in fact
50apparently settled down immovably.
51By some curious chance one morning long ago in the quiet of the world, when there was
52less noise and more green, and the hobbits were still numerous and prosperous, and Bilbo
53Baggins was standing at his door after breakfast smoking an enormous long wooden pipe that
54reached nearly down to his woolly toes (neatly brushed) Gandalf came by. Gandalf! If you had
55heard only a quarter of what I have heard about him, and I have only heard very little of all
56there is to hear, you would be prepared for any sort I of remarkable tale. Tales and adventures
57sprouted up all over the place wherever he went, in the most extraordinary fashion. He had not
58been down that way under The Hill for ages and ages, not since his friend the Old Took died, in
59fact, and the hobbits had almost forgotten what he looked like. He had been away over The Hill
60and across The Water on business of his own since they were all small hobbitboys and
61hobbitgirls.
Output
1[
2 {
3 "node_1": "hobbit",
4 "node_2": "hobbit-hole",
5 "edge": "A hobbit lives in a hobbit-hole."
6 },
7 {
8 "node_1": "hobbit-hole",
9 "node_2": "tunnel-shaped hall",
10 "edge": "A hobbit-hole has a tunnel-shaped hall."
11 },
12 {
13 "node_1": "hobbit-hole",
14 "node_2": "The Hill",
15 "edge": "A hobbit-hole is located in The Hill."
16 },
17 {
18 "node_1": "hobbit",
19 "node_2": "Bilbo Baggins",
20 "edge": "Bilbo Baggins is a hobbit."
21 },
22 {
23 "node_1": "hobbit",
24 "node_2": "Bungo Baggins",
25 "edge": "Bungo Baggins is a hobbit and Bilbo Baggins' father."
26 },
27 {
28 "node_1": "hobbit",
29 "node_2": "Belladonna Took",
30 "edge": "Bilbo Baggins' mother is Belladonna Took."
31 },
32 {
33 "node_1": "hobbit",
34 "node_2": "Gandalf",
35 "edge": "Gandalf meets Bilbo Baggins."
36 },
37 {
38 "node_1": "hobbit",
39 "node_2": "Old Took",
40 "edge": "The Tooks are hobbits and ancestors of Bilbo Baggins."
41 },
42 {
43 "node_1": "hobbit",
44 "node_2": "Bagginses",
45 "edge": "The Bagginses are a hobbit family."
46 },
47 {
48 "node_1": "hobbit",
49 "node_2": "Tooks",
50 "edge": "The Tooks are a hobbit family."
51 },
52 {
53 "node_1": "hobbit-hole",
54 "node_2": "luxurious",
55 "edge": "Bilbo Baggins' hobbit-hole is luxurious."
56 },
57 {
58 "node_1": "hobbit-hole",
59 "node_2": "comfortable",
60 "edge": "Bilbo Baggins' hobbit-hole is comfortable."
61 },
62 {
63 "node_1": "hobbit-hole",
64 "node_2": "panelled walls",
65 "edge": "Bilbo Baggins' hobbit-hole has panelled walls."
66 },
67 {
68 "node_1": "hobbit-hole",
69 "node_2": "tiled and carpeted floors",
70 "edge": "Bilbo Baggins' hobbit-hole has tiled and carpeted floors."
71 },
72 {
73 "node_1": "hobbit-hole",
74 "node_2": "polished chairs",
75 "edge": "Bilbo Baggins' hobbit-hole has polished chairs."
76 },
77 {
78 "node_1": "hobbit-hole",
79 "node_2": "lots of pegs",
80 "edge": "Bilbo Baggins' hobbit-hole has lots of pegs."
81 },
82 {
83 "node_1": "hobbit-hole",
84 "node_2": "round doors",
85 "edge": "Bilbo Baggins' hobbit-hole has round doors."
86 },
87 {
88 "node_1": "hobbit-hole",
89 "node_2": "porthole-shaped door",
90 "edge": "Bilbo Baggins' hobbit-hole has a porthole-shaped door."
91 },
92 {
93 "node_1": "hobbit-hole",
94 "node_2": "green door",
95 "edge": "Bilbo Baggins' hobbit-hole has a green door."
96 },
97 {
98 "node_1": "hobbit-hole",
99 "node_2": "yellow brass knob",
100 "edge": "Bilbo Baggins' hobbit-hole has a yellow brass knob."
101 },
102 {
103 "node_1": "hobbit-hole",
104 "node_2": "tube-shaped hall",
105 "edge": "Bilbo Baggins' hobbit-hole has a tube-shaped hall."
106 },
107 {
108 "node_1": "hobbit-hole",
109 "node_2": "passage",
110 "edge": "Bilbo Baggins' hobbit-hole has a passage."
111 },
112 {
113 "node_1": "hobbit-hole",
114 "node_2": "bedrooms",
115 "edge": "Bilbo Baggins' hobbit-hole has bedrooms."
116 },
117 {
118 "node_1": "hobbit-hole",
119 "node_2": "bathrooms",
120 "edge": "Bilbo Baggins' hobbit-hole has bathrooms."
121 },
122 {
123 "node_1": "hobbit-hole",
124 "node_2": "cellars",
125 "edge": "Bilbo Baggins' hobbit-hole has cellars."
126 },
127 {
128 "node_1": "hobbit-hole",
129 "node_2": "pantries",
130 "edge": "Bilbo Baggins' hobbit-hole has pantries."
131 },
132 {
133 "node_1": "hobbit-hole",
134 "node_2": "wardrobes",
135 "edge": "Bilbo Baggins' hobbit-hole has wardrobes."
136 },
137 {
138 "node_1": "hobbit-hole",
139 "node_2": "kitchens",
140 "edge": "Bilbo Baggins' hobbit-hole has kitchens."
141 },
142 {
143 "node_1": "hobbit-hole",
144 "node_2": "dining-rooms",
145 "edge": "Bilbo Baggins' hobbit-hole has dining-rooms."
146 },
147 {
148 "node_1": "hobbit",
149 "node_2": "well-to-do",
150 "edge": "Bilbo Baggins is well-to-do."
151 },
152 {
153 "node_1": "hobbit",
154 "node_2": "respectable",
155 "edge": "Bilbo Baggins is considered respectable."
156 },
157 {
158 "node_1": "hobbit",
159 "node_2": "adventures",
160 "edge": "Bilbo Baggins has adventures."
161 },
162 {
163 "node_1": "hobbit",
164 "node_2": "unexpected things",
165 "edge": "Bilbo Baggins does unexpected things."
166 },
167 {
168 "node_1": "hobbit",
169 "node_2": "Gandalf",
170 "edge": "Bilbo Baggins meets Gandalf."
171 },
172 {
173 "node_1": "hobbit",
174 "node_2": "Big People",
175 "edge": "Hobbits consider Big People as large and stupid."
176 },
177 {
178 "node_1": "hobbit",
179 "node_2": "elephants",
180 "edge": "Hobbits can hear elephants coming from a mile away."
181 },
182 {
183 "node_1": "hobbit",
184 "node_2": "green and yellow",
185 "edge": "Hobbits dress in bright green and yellow colors."
186 },
187 {
188 "node_1": "hobbit",
189 "node_2": "no shoes",
190 "edge": "Hobbits do not wear shoes."
191 },
192 {
193 "node_1": "hobbit",
194 "node_2": "natural leathery soles",
195 "edge": "Hobbits have natural leathery soles."
196 },
197 {
198 "node_1": "hobbit",
199 "node_2": "thick warm brown hair",
200 "edge": "Hobbits have thick warm brown hair."
201 },
202 {
203 "node_1": "hobbit",
204 "node_2": "long clever brown fingers",
205 "edge": "Hobbits have long clever brown fingers."
206 },
207 {
208 "node_1": "hobbit",
209 "node_2": "good-natured faces",
210 "edge": "Hobbits have good-natured faces."
211 },
212 {
213 "node_1": "hobbit",
214 "node_2": "deep fruity laughs",
215 "edge": "Hobbits laugh deeply and fruitily."
216 },
217 {
218 "node_1": "hobbit",
219 "node_2": "Bagginses",
220 "edge": "The Bagginses are a hobbit family."
221 },
222 {
223 "node_1": "hobbit",
224 "node_2": "Tooks",
225 "edge": "The Tooks are a hobbit family."
226 },
227 {
228 "node_1": "hobbit",
229 "node_2": "Old Took",
230 "edge": "The Old Took is a hobbit and ancestor of Bilbo Baggins."
231 },
232 {
233 "node_1": "hobbit",
234 "node_2": "fairy wife",
235 "edge": "It is said that one of the Took ancestors took a fairy wife."
236 },
237 {
238 "node_1": "hobbit",
239 "node_2": "adventures",
240 "edge": "The Tooks have adventures."
241 },
242 {
243 "node_1": "hobbit",
244 "node_2": "disappear quietly and quickly",
245 "edge": "Hobbits can disappear quietly and quickly."
246 },
247 {
248 "node_1": "hobbit",
249 "node_2": "stupid folk",
250 "edge": "Hobbits consider Big People as stupid."
251 },
252 {
253 "node_1": "hobbit",
254 "node_2": "elephants",
255 "edge": "Hobbits can hear elephants coming from a mile away."
256 },
257 {
258 "node_1": "hobbit",
259 "node_2": "green and yellow",
260 "edge": "Hobbits dress in bright green and yellow colors."
261 },
262 {
263 "node_1": "hobbit",
264 "node_2": "no shoes",
265 "edge": "Hobbits do not wear shoes."
266 },
267 {
268 "node_1": "hobbit",
269 "node_2": "natural leathery soles",
270 "edge": "Hobbits have natural leathery soles."
271 },
272 {
273 "node_1": "hobbit",
274 "node_2": "thick warm brown hair",
275 "edge": "Hobbits have thick warm brown hair."
276 },
277 {
278 "node_1": "hobbit",
279 "node_2": "long clever brown fingers",
280 "edge": "Hobbits have long clever brown fingers."
281 },
282 {
283 "node_1": "hobbit",
284 "node_2": "good-natured faces",
285 "edge": "Hobbits have good-natured faces."
286 },
287 {
288 "node_1": "hobbit",
289 "node_2": "deep fruity laughs",
290 "edge": "Hobbits laugh deeply and fruitily."
291 },
292 {
293 "node_1": "hobbit",
294 "node_2": "The Hill",
295 "edge": "The Hill is the location of Bilbo Baggins' hobbit-hole."
296 },
297 {
298 "node_1": "hobbit",
299 "node_2": "quiet of the world",
300 "edge": "The world was quieter in the past."
301 },
302 {
303 "node_1": "hobbit",
304 "node_2": "less noise",
305 "edge": "There was less noise in the world in the past."
306 },
307 {
308 "node_1": "hobbit",
309 "node_2": "more green",
310 "edge": "The world was greener in the past."
311 },
312 {
313 "node_1": "hobbit",
314 "node_2": "hobbits were numerous and prosperous",
315 "edge": "Hobbits were numerous and prosperous in the past."
316 }
317]
Visualize Graph
This script assumes graph.json
exists in the same directory.
Install dependencies
1pip install networkx
1pip install pyvis
1import json
2import os
3import random
4from pyvis.network import Network
5import networkx as nx
6
7def generate_random_color():
8 """Generate a random hex color."""
9 return "#{:06x}".format(random.randint(0, 0xFFFFFF))
10
11# Path to the JSON file
12file_path = os.path.join(os.path.dirname(__file__), 'graph.json')
13
14# Load data from JSON file
15with open(file_path, 'r') as file:
16 data = json.load(file)
17
18# Create a NetworkX graph
19G = nx.Graph()
20
21# Add nodes and edges from the data
22for item in data:
23 G.add_node(item["node_1"])
24 G.add_node(item["node_2"])
25 G.add_edge(item["node_1"], item["node_2"], title=item["edge"])
26
27# Detect communities
28communities_generator = nx.community.girvan_newman(G)
29top_level_communities = next(communities_generator)
30next_level_communities = next(communities_generator)
31communities = sorted(map(sorted, next_level_communities))
32
33# Create a PyVis network from the NetworkX graph
34net = Network(height="1100px", width='100%', notebook=False, select_menu=True)
35
36# Assign different random colors to different communities
37for community in communities:
38 color = generate_random_color()
39 for node in community:
40 net.add_node(node, node, color=color)
41
42# Add edges
43for edge in G.edges(data=True):
44 net.add_edge(edge[0], edge[1], title=edge[2]['title'])
45
46# Add additional options for better visualization
47net.set_options("""
48var options = {
49 "nodes": {
50 "font": {
51 "size": 14
52 }
53 },
54 "edges": {
55 "color": {
56 "inherit": true
57 },
58 "smooth": false
59 },
60 "physics": {
61 "barnesHut": {
62 "gravitationalConstant": -80000,
63 "centralGravity": 0.3,
64 "springLength": 200,
65 "springConstant": 0.04,
66 "damping": 0.09,
67 "avoidOverlap": 0.1
68 },
69 "minVelocity": 0.75
70 }
71}
72""")
73
74# Save the network
75#net.show("network.html")
76net.write_html("network.html", local=True, notebook=False, open_browser=True)