2014 goto

goto 2014 Slide Download

Martin Fowler: Software Development in the 21st Century

http://martinfowler.com

http://martinfowler.com/tags/talk%20videos.html

Microservices

  • Definition? When to use?
  • monolith vs. microservices
    • scaling
  • Definition of Microservices
    • Better: common characteristics
      • http://martinfowler.com/articles/microservices.html
        1. Componentization via services

          • Organized around business capabilities

          • in-dependably maintainable & upgradable
          • Component: Library vs. Service
            • Library has tighter coupling
        2. Organized around Business Capabilities

        3. Products not Projects (organization)
        4. Smart Endpoints and dumb pipes
          • ESB vs. dumb pipes
        5. Decentralization in Governance
        6. Decentralization in Data Management
          • also like Amazon
          • every service should have it's own datastore
          • Services may only talk through API, never through a database
        7. Infrastructure Automation
        8. Design for Failure
          • Chaos Monkey
          • Monitoring
        9. Evolutionary Design
  • Are Services SOA?
    • means different things to different people
    • Microservices subset of SOA
  • How Big?
    • small enough to fit in ones head
  • When to use?
MonolithMicroservice
  • simplicity (up to a certain size)
  • consistency (microservices enforce eventual consistency)
  • Inter-module refactoring
  • Partial Deployment
  • availability
  • preserve modularity
  • multiple plaforms
  • be sure to figure out clear boundaries
  • must haves:
    • rapid provisioning
      • e.g. cloud
    • basic Monitoring
    • rapid application deployment
    • devops culture
    • ...

other talk / not just code monkeys

  • Waterfall: Spec => Software
  • Agile: break things down
    • Feedback
    • Visible Process
    • Steering
    • (warning) Problem:
      • dev team passive recipient of stories
        • "engine to turn stories into code"
        • better "Conversational Stories"
          • come up with better ideas
            • taking technology into account
            • proved by monitoring / tracking
          • Knowledge
            • developers need to get deep into that
            • more important than technical knowledge
  • Dark Patterns
    • http://darkpatterns.org/
    • manipulating a user to do something that is not in his best interest
    • Developers need to be advocates for the users
      • MF: "We are responsible for the software we write"
      • "I just follow orders" not good enough!
  • Impact Judgement
    • Do you do something that gives value to the world?
    • MF: "Where do we want to apply our talents?"
    • MF: "Make the world a better place"
  • Improvement needed
    1. Alienating Atmosphere in IT world
    2. Privacy
      • persecution by governance
      • criminal groups
  • take responsibilities
  • engage with the world
    • show that we are not just strange guys in cellars
  • act as leaders

Simple Scalability - Microservices on PaaS (Colin Humphreys)

  • ColinHumphreys_SimpleScalabilityMicroservicesOnPaaS.pdf
  • @hatofmonkeys
  • Patterns
    • organizational
      • teams focusing on a functionality
      • authorative PO
        • what ware we doing?
        • when is it done?
      • T-shaped people
      • everybody a a speciality
      • understand role in value stream
        • where to fit to maximize productivity
      • Domain driven Design
      • Process (no matter if scrum, xp or waterfall)
        • Kickoff meeting
        • all stakeholders
        • fill backlog
        • Iterative planning meeting
        • retrospective
    • architectural
      • nouns & verbs
        • verbs for services
      • create pipelines of microservices
        • tolerant readers
      • Communication Strategies
        • Sync: https/json
        • Async: Msg Q JSON (e.g. redis / rabbit MQ)
        • Design for failure
        • Circuit Breakers
      • Data
        • Immutable idempotent event
        • no matter how often you replay these events, they are still true
        • SQRS: separating commands from queries
        • 12 factor application (http://12factor.net/)
        • PaaS Compatible
        • no state in application
      • Let microservices die
        • when responsibility is obsolete: remove microservice
        • evolutionary, darwinism
    • Operational
      • Blue/green deployment
      • Shock absorber
        • queues
      • dynamic Scaling
      • Instrumentation
        • trace ID through system
          • output to logging
          • centralize logs (e.g. elastic search / logstash)
  • http://en.wikipedia.org/wiki/Polyglot_(computing)
  • Cloud Foundry

Craftmanship over Hype

  • EnriqueCombaRiepenhausen_CraftsmanshipOverHype.pdf
  • Enrique Combat Riepenhausen
    • value-based charging
    • patheleven outdoors programming
    • "best restaurant"
      • my opinion: high price!!!
        • "elite" software
    • Book called "Inceptions - Starting a Software Project"
      • ECR: "You need to learn the domain"
    • Makers Acedemy
    • Can you teach somebody to code in 12 weeks?
  • responsibility for what you program
  • Summit "Software Craftsmanship" London
    • Manifesto for Software Craftmanship
    • 14935 signatories so far
    • demonization
      • code being attacked
  • well crafted software
    • testing
      • does not always make sense
    • "Craftsmanship names an enduring, basic human impulse, the desire to do a job well for its own sake"
  • Steadily adding value
    • challenge, ask why
    • "To asses the quality of thoughts of people, don't listen to their words, but watch their actions"
  • Community of professionals
    • exchange inside information
    • exchange people
  • Productive Partnerships
  • "We are what we repeatedly do. Excellence then, ist not an act, but an habit."
  • "Wherever you go becomes a part of you somehow." - Anita Desai
  • ECR "Why should you care to Standup and go to your work?"

Towards Cloud-first Development

  • PetervanHardenberg_TowardsCloudFirstDevelopment.pdf
  • Peter van Hardenberg
    • heroku
      • platform-as-a-service
      • "machine for turning code into running applications"
  • "If I asked the People what hey want, they would have told me faster horses"
  • "I din't always test my code … but if I do, I do it in production"
  • fourchette
    • "What if you could automatically test in production?"
  • work for team
    • provisioning
    • release process
    • monitoring
    • capacity management / scale
    • substantial per app cost
    • Conclusion: reduce costs by outsource to PaaS provider
  • Cloud / PaaS
    • individual applications approach zero overhead
    • microservice achitectures become feasible
    • Clone whole running applications
    • Cinclusion: decompose application achitecture
  • "Microservices love PaaS"
  • postgression
  • the hidden costs of releasing software
    • Cumulative cost
    • cost of big releases
      • Big changes are costlier than small changes
    • amazon:
      • each time a server is down, it is replaced with a new one
    • don't fix servers - they're free in PaaS
    • Treat you servers like cattle, not pets. => Don't get attached to servers
  • Why consume cloud services?
    • own dynamo vs. power plant
    • Who has to fix problems? 24/7?
    • amazon has hundrets of thousands of servers
  • Who serves your usecase best?
    • Service-based businesses get paid, when you are happy
    • Consultancies get paid when you are unhappy
    • Great Services, like great consultancies, focus on making your project successful
  • Opportunites (not used yet)
    • programming environments as a service
      • setting up & maintaining laptop dev/statging/prod environment is very error prone
    • release orchestration (automated)
    • continuous disintegration (?)
    • circuit brakers as a service
      • better failure modes for modules
  • Think of somethings horrible in development
    • What if it went away? What new thing could do?
  • Technology provided on demand create new opportunities
  • Tradeoff: Automating a problem vs. cost of automating a problem
  • Feed of the creative ideas of younger developers

Graph All The Things!! Graph Database Use Cases That Aren't Social

Our Responsibility to Defeat Mass Surveillance

  • ErikDoernenburg_and_MartinFowler_OurResponsibilityToDefeatMassSurveillance.pdf
  • Martin Fowler, Erik Doernenburg
  • 1984 - excerpt
  • todays monitoring: voice, localtion tracking, …
  • some people say: I don't mind living in a glass house
    • => actually not true for most ("shit in the woods etc" ;-) )
  • "I have nothing to hide"
  • "Nobody is interested in watching me"
  • A lot of the stuff is not about the I and the me
  • MF: "Activists change society"
  • MF: "Heard immunity is something that we need"
  • https://www.tbray.org/ongoing/When/201x/2014/05/26/Privacy-Levels
    • Basic Privacy: Stop random strangers watching you (https, encrypted wifi) - $
    • Common Privacy: Stop crooks, Government needs warrant (process for protection needed) - $$$$$$$$$$
      • does not work in virtual world
        • borderless
          • do we want other governments to see our stuff?
    • Strong Privacy: Stop mostly everyone - $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
      • raise cost of breaking privacy
      • becomes harder to target bothersome people
  • free society <= Free Association and Communication
  • Examples
    • Email
      • Maximize shareholder value 
      • conflict between customers and shareholder
        • drops on the developers
      • Everybody say: it's not my problem, somebody else is gonna do it
        • "I need the money" is no valid argument for developers
      • Trend towards SaaS / webmail
        • Not many small providers
        • market share of the "big 5" email providers globally
          • ~ 42% Gmail, Yahoo, Hotmail etc.
          • rather easy to get access to 5 US companies
        • Gemany: top 8: 88%
        • US: top 4: 89%
      • As long as you are a US company, you have to reveal data to US state, even if it is hosted in Europe
        • currently the juridical practice
    • "I have nothing to hide"
      • opaque price (you can not compare prices any more)
      • Jeremy Hammond
        • 10 Years in Jail for hacking Stratfor
          • Stratfor can access classified FBI information
      • seeping of protected data into the commercial sector
        • example: Health Insurance, Cancer-example in Email-Communication
  • pixelated
    • Secure Email System
    • goals:
      • Mass adoption
      • Open Source
      • Deliver Software Product
      • Context of "broken internet"
      • UX: as convenient as current solutions
    • even with PGP a lot of information is available
      • discover social network
    • Pixelated Platform:
      • encrypted storage
      • encrypted emails
      • private keys on server
  • What do you do if you discover dark patterns?
    • Decision:
      1. It' OK! ($$$$)
      2. Won't do it as it' unethical (probably $$)
    • WE have to make this decision
      • Being responsible over maximizing gain

Morning Keynote: Excellence Culture & Humane Keeping of Techies

  • GunterDueck_MorningKeynoteExcellenceCultureHumaneKeepingOfTechies.pdf
  • Gunter Dueck
    • Buch Professionelle Intelligenz
  • Who moved my Cheese
    • (Geman) obsessive compulsive mouse vs. (American) hysterical mouse
    • Leave comfort zone and find cheese :-D
  • Who morphed my cheese
    • mice look for cheese, but there's only bacon, so they die :-)
  • Replacement of humans by automization
    • solves some problems (self driving cars and with this energy problems etc.)
    • kills a lot of jobs
  • Work hard to stay to the top - by much more frequent and quicker …..
  • "normal" is to expensive
    • upgrade to master or
    • be badly paid "slave"
  • T-Shaped: Breadth of Knowledge, Depth of Expertise
    • Wide: Influence spread widely across the group or organization. Important for knowing issues, having things on the radar, building wide acceptance and buy-in
    • Focused: Influence concentrated with one person ... team ….
  • Futre Key Competencies
    • Society / Mother wants:
      • study hard
      • keep everything in order
      • behave well
      • cooperate
    • Future
      • ...
      • take Autism Quotion (Homepage)
      • => get selling competence (bacon)
      • Customer complaints: Social competence missing
      • Professional Intelligence
        • IQ - structure, Plan, Organization
        • EQ - Understanding, & "Team" (Empathy)
        • CQ - Creativity, Openness, Curiosity
        • AQ - Talent for attraction ("intropathy" - make feelings in you opposite, that are not there)
        • VQ - Vitality, Will Power, Energy, Passion
        • MQ - Sense for Meaning
        • PQ - Integrated Professionally
  • Clash of cultures
    • how to heard cats?
    • Cats want to make Dogs understand, that they are wrong - but the dogs have the power and the money
    • You have to have some empathy with dogs
  • Book: Spiral Dynamics
    • Germany: blue
    • America: orange
    • Sweden: green
    • Technical people: yellow
  • How to reward a cat?
    • Conferences
    • Climb the ladder: work vor innovation, customer relations and sustainability

Handling Data-Flow the Reactive Way using Akka Streams

  • RolandKuhn_HandlingDataFlowTheReactiveWayUsingAkkaStreams.pdf
  • Roland Kuhn
  • Why Akka Streams?
    • if you can't fit your data in the memory, put you want to process it
    • real-time data processing (CEP)
    • serving numerous clients simultaneously with bounded resources (IoT, streaming HTTP APIs)
  • What is a stream?
    • ephemeral, time-dependent sequence of elements
    • possibly unbounded in length
    • therefore focusing on transformations
    • "You cannot step twice into the same stream. For as you are stepping in, other waters are ever flowing on to you"
  • Declaring a Stream Topology
    • Source (something that as an output of a certain type)
    • Partial Flow Graph: [ (s1, s2) => merge, zip => ]
      • Is a source, because it has one output
    • Source -> Flow -> Sink
      • Source: 1 output
      • Flow: 1 in, 1 out
      • Sink: 1 n
    • -code example-
    • Akka Streams separate the what from the how
      • this allows alternative materialization strategies
    • Stream Source examples: Collections, Futures, Publisher/Subscriber, http connections ...
    • Stream Sink examples: Publisher, Subscriber, ActorSubscriber, http connections …
    • Linear Stream Transformations
      • deterministic
        • map, filter, collect, …
      • Time-based
        • takeWithin, dropWithin, groupedWithin, …
      • Rate-Detached
        • expand - how to make up missing values
        • conflate - select how to aggregate things
        • buffer
      • Async
        • mapAsync: map that returns future
        • mapAsyncUnordered: unordered variant of above
        • flatten: stream of streams => stream
      • Nonlinear Stream transformation
        • Fan-In
        • Fan-Out
  • Reactive Traits
    • Responsive, elastic, resilient, message-driven
  • => Back-Pressure / Flow Control
    • the reactive streams initiative (input of Netflix, Oracle, Pivotal, Twitter etc.)
    • common solution would help everyone
  • recipe for success
    • minimal interfaces
    • rigorous specification of semantics
    • full TCK for verification of implementation
    • complete freedom for many idiomatic APIs
  • API
    • Subscriber get Subscription
    • Subscription is used to tell the publisher that it can consume more via request(n)
  • Supply and Deman
    • data items flow downstream
    • demand flows upstream
    • data items flow only when there is demand
      • recipient is in controll
  • Dynamic Push-Pull

 

Notes

  • java one talk über akka persistence anschauen!
  • reactive manifesto anschauen
  • Actor Frameworks anschauen

Martin Fowler "special" 

  • automation of deployment very important
  • monitoring essential
  • probably better many small repository
    • Google: one big repo
    • unproven yet
  • Versioning API: better don't do it!
  • Postals Laws (?): be strict on what you give, and tolerant in what you accept
  • Soon an article on testing on Martin Fowlers page
  • Testing:
    • Use many unit tests
    • Use contract testing: pact

Soundcloud Microservices

  • PhilCalcado_WhatHasSoundCloudLearntAboutMicroservices.pdf
  • http://philcalcado.com
  • "Youtube for Audio"
    • 11h of audo upload per minute
    • 300 mio people very month
  • http://bit.ly/dealing-with-the-monolith
  • minimise you fixed (upfront) cost per app 
  • Every new app needs answers
    • How to minimize this?
  • Technical Task decrease over iterations
    • for Microservices:
      • this repeats over an over!
  • even Soundcloud consolidated languages
    • define a well-supported and documented standard-stack
      • currently @Soundcloud: scala, memcache-d, redis, rabbit-mq
  • outsource as much as possible
    • not people!
    • Infrastructure
      • spend majority of time on app
      • spend minimal time on infrastructure
  • Frameworks
    • finagle Framework
      • works best with Scala
    • Dropwizard
      • java
    • Netflix Oss
      • check block post
  • avoid having quirks
    • do things in the best interest of the organization
  • acknowledge the fact that you have a new complexity
    • shaped in a different way
      • many match organizational structure better
    • monolith: you could run some things in unit tests
      • in microservice platform you might need integration tests, or mocked component tests
  • 12-Factor App + docker + fig
    • fig also in production?
  • make everything visible
    • how do you get communication between (service) teams?
    • zipkin:
      • firebug meets micro services
    • standardized dashboards
      • visible to everyone (over url)
      • possibility of snapshots
  • wrap you legacy with stranglers
    • StranglerApplication by Martin Fowler
  • Soundcloud:
    • created internal API => kickoff
    • enable & empower app devs
    • experience based api?
  • RPC instead of REST?
  • Thrift?
  • ephemeral test environments

 

Notes

  • wir sollten auch mal über einen standard-stack nachdenken
    • aber erstmal was ausprobieren!
  • Martin Fowler Blog, PayPal & Soundcloud Blog abonnieren
    • + den vom Groovy Typen
    • InnoQ?
  • 12-Factor App anschauen! (Heroku philosophy for applications)
  • What to use to orchestrate micro services?
  • check zipkin
  • StranglerApplication by Martin Fowler anschauen
    • Strangler für CS :-) ?
  • Check InnoQ Talk: circuit breaker for play
  • finagle anschauen

Scaling Fashionably: From Startup to Scale at Zalando

  • PhilCalcado_WhatHasSoundCloudLearntAboutMicroservices.pdf
  • Valentine Gogichashvili
  • Postgres
  • SOLR
  • Zalando
    • challenges
      • constantly growing
      • fast development cycles
      • no downtimes tolerated
  • access data
    • NoSql
      • pro
        • map your objects hiarachy to a docuemtns
        • serialization is easy
        • no transcation needed
      • problems
        • NO SQL / not possible to change only small parts
        • implicit data schemas are tricky
          • schema is stored in application logic
      • OMR
        • pro
          • well known to developers
          • CRUD operations easy
          • business logic inside your application
          • developers are in the comfort zone
        • problems
          • error prone transaction management
          • you have to reflect your tables in you code
          • business logic inside your application
          • schema changes not easy
      • stored procedures
        • pros
          • return / receive entity aggregates
          • clear transaction scope
          • more data consistency checks (close to data)
          • independent from underlying data schema (???)
          • use-case driven
        • layout:
          • java
          • SProc Wrapper @SProcCalls
          • jdbc
          • stored procedure api
          • database table
        • problems
          • CRUD operations need too much code
          • Developers have to learn SQL
          • Developers can write bad SQL
          • Debugging
        • versioning
          • schema api_v13_01
          • new schema will be installed in parallel
          • database table changes need to be downward compatible
          • pros:
            • Tests are done to the whole API version
            • No API migration is needed
            • Deployments are fully automated (?)
      • tipp: learn functional programming
  • change data models without downtime
    • easy schema changes
      • Schema changes with minimal locks:
        • add / rename / drop column
        • add/drop default value
        • create/drop index concurrently
      • Constraints are still difficult to alter
      • Stored Procedure API Layer
        • can fill missing data on the fly
        • helps to change data structure without applcation noticing it
      • SQL Diff scripts written by Java developers
      • Db-Diff tool
  • shard without limits
    • One big database
      • data does not fit into memory => Disk access
      • OLTP becomes slower
      • Longer data migration times
      • database maintenance tasks take longer
      • (no scaling!?)
    • Shared databse
        • data fits into memory
        • IO bottlenecks are wieder
        • OLTP ist fast again
        • data migrations are faster
      • cons:
        • you can not join between entity aggregates
        • BI needs more tooling
        • accessing data needs more tooling
      • @Zalando:
        • Shard mapping in SprocMapper
        • search on all shards …
        • unique shard aware id
        • idea from tumbler
  • monitoring postgres DBs
    • pg_view
    • create small helpers to access shards
    • replaced nagios/icinga by ZMON2 (custom built)
    • self-written tool PGObserver (open source project)
      • predicts destabilities
  • download slides for links

Notes

  • learn functional programming
  • create staging environment for each push
  • was ist tumbler?
  • DB-experten?

Fun stories from an architects life

  • StefanTilkov_FunStoriesFromAnArchitectsLife.pdf
  • Stefan Tilkov
  • project example 1: less is more
    • environment:
      • Bing bang
      • customer care, accounting & billing
      • large-scale big-bang replacement effort
      • > 150 java developers
      • > 20 architects/framework devs
      • Java, Swing, CORBA, Oracle
    • Custom persistence framework
      • cache in OR-mapper
    • Removal of cache made the system faster (because oracle cache was much better)
    • Lessons learned:
      • don't cache the cache
      • application state is your enemy
      • if it's "sophisticated" & fun, you probably shouldn't be doing that
      • measure in production
  • project example 2
    • insurance
    • smalltalk, cobol
    • lessons learned:
      • data should be free of code dependencies
      • the longer you store it, the simpler it needs to be
      • Beware of smelly blobs
  • project example 3: Evaluate yourself to death
    • which application server to use?
      • vendor selection: 26 Week
    • lessons learned
      • Just do it (may be the wrong one, but you can start today)
      • Better ask for forgiveness than permission
  • project example 4: Your System will be dynamic
    • large-scale insurance system
    • model-driven development
    • 2 Releases per year
    • 2-week modeling slot
      • if missed: wait for one year
    • Excel-sheet mapping between real rows and designed rows
    • lessons learned
      • centralized responsibility hurts
      • faced with too much rigidity, a way around the rules will be found
      • Just because you're used to doing it, doesn't make it acceptable
  • Project example 5
    • Telecommunications OSS Platform
    • support new devices without code changes
    • lessons learned
      • if it makes you want to scratch your eyes out, you might not want to do it => shout otu!
      • sometimes constraints need to be fought
      • beware all things "meta"
  • Project example 6: The Perils of being smart
    • enterprise-wide SOA effort
    • build own middleware
    • lessons learned
      • fight the madness
      • Tunneling is the root of all evil (tunneling = trying to turn something in something that it isn't)
      • Don't be too smart
  • Project example 7: Non-extensible Extensibility
    • E-Commerce
    • Catalog/CMS/Shop/Fulfillment
    • multi tended
    • lessons learned
      • don't be all things to all people
      • Sometimes specific code is preferable
      • Saying "no" might help (let's do that later :-P )
  • Summary
    • never-ending supply of stories
    • advice:
      • Give Feedback
      • Reflect on what you do (again and again)

Continuous delivery and devops in large enterprises 

  • EberhardWolff_DevOpsAndContinuousDeliveryInTheEnterprise.pdf
  • Jonathan Weiss
    • Buch Continuous Delivery
  • Buch: Eric Ries: The Lean Startup
    • validated learning
    • scientific experimentation
    • iterative product releases
    • shorten product development cycles
    • measure progress
    • gain valuable customer feedback about features
  • Observe, Orient, Decide, Act (OODA) - Loop


EA Lessons Learned the Hard Way

  • AmandaLaucher_EALessonsLearnedTheHardWay.pdf
  • Amanda Laucher
  • Innovation vs Standardization
  • hardest challenge: sizing
  • employ microservices only where necessary
  • Deployment Model
  • Immutable Infrastructure
  • Naming is a hard problem
  • schema-less sucks :-)
  • documentation is important
  • Model put spotlights on specific parts, you can not put every detail into it
  • System will mimic the organization in which they are built
  • employ new technologies only if they solve problems
  • Know what you don't know
  • Booklist:
    • The Goal and Beyond the Goal
    • Predictably Irrational
    • Thinking in Systems
    • …. 


Notes

  • MarketFinder , ProductFinder, MarketImporter, ProductImporter
  • Market - Price - Product
  • PriceFinder?
  • Talk to Thomas, Finies, Alwin (auch wegen Versionierung)

 

Market with products and prices

  • + not much overhead
  • + fits the image of a market
  • + high coupling between Markets and products
  • ~ probably already a bit too big to fit into ones head?
  • ~ Which is the separation criterium? Market could contain all
    • - not well understandable

 

MarketFinder, ProductFinder, PriceFinder with separate SOLR

  • + Composability
  • + Specific scaling
  • - how too keep data in sync (new market, prices have not been updated)
  • - too much overhead, does not really make sense

 

MarketFinder, ProductFinder with shared SOLR

  • + high coupling between Markets and products
  • + Understandability
  • + no sync problems

 


Notes

  • don't forget Circuit Braker Pattern
  • Kai: more devops for microservice architecture (requirement as to MF)
  • Markets & Products
    • share one component
      • easy read-only format conversion
      • high coupling between Markets & products
        • Prices
        • Offers
        • "Bestand"
      • can be decoupled when needed
  • Have a look at java one talk (which one??)
  • Talk to Robert about scala
  • try PaaS for us!
  • read manifesto :-)
  • let our code be reviewed by commerce tools
  • make programming be real fun
    • How to teach that?
  • use cloud foundry
  • use verbs as basis for services